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Executive  Summary 


ASSIP 


This  survey 


Survey 

participants 


Objectives 


Scope 


Interpreting  the 
results 


Major  Theme 


The  U.S.  Army  Strategic  Software  Improvement  Program  is  referred  to 
throughout  this  report  as  ASSIP. 

The  ASSIP  is  a  long-term  effort  focusing  on  acquisition  programs,  people, 
production  and  sustainment,  and  the  institutionalization  of  continuous 
improvement. 


This  study  was  conducted  by  the  Software  Engineering  Institute  (SEI)on 
behalf  of  ASSIP.  The  survey  (whose  results  are  reported  herein)  is  one  of 
several  information-gathering  approaches  to  support  effective  decision 
making  within  the  ASSIP. 


The  intended  audience  targeted  for  participation  in  this  survey  are  Army 
program  managers  who  are  responsible  for  the  acquisition  of  software¬ 
intensive  systems.  A  total  of  150  individuals  participated  in  this  survey. 


The  objectives  of  this  initial  survey  were  to 

•  provide  preliminary  insight  into  the  major  acquisition-related  problem 
areas  (as  perceived  by  Army  program  managers)  so  that 
improvement  opportunities  can  be  prioritized 

•  assist  planning  of  future  data  collection  activities  by  shedding  light  on 
the  problem  space,  thereby  exposing  areas  that  should  be  the  object 
of  more  detailed  analysis  as  part  of  future  data  gathering  efforts 


In  this  survey,  the  researchers  were  requested  to  cover  four  areas  of  the 
acquisition  system  including 

1 .  the  acquirer’s  environment 

2.  communication  between  the  acquirer  and  developer 

3.  the  developer’s  environment 

4.  factors  that  are  external  to  the  Acquisition  System  (but  may  impact  it) 

This  survey  is  only  one  part  of  an  overall  data-gathering  approach  to 
understand  the  state  of  the  Army  Acquisition  System.  Therefore,  the  results 
should  not  be  viewed  as  conclusive,  but  should  be  used  with  other  sources  of 
information  for  proper  interpretation.  Due  to  sampling  issues  (explained  in  the 
body  of  this  report),  results  cannot  be  generalized  to  the  overall  population  of 
Army  Acquisition  Managers. 


Requirements  management  is  a  primary  area  of  concern  to  Army  acquisition 
program  managers  who  responded  to  this  survey. 

Other  areas  that  should  be  further  explored  as  potential  areas  for  high-impact 
improvement  include 
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•  unstable  funding  •  risk  management 

•  soiicitation  •  transition-to-support 

•  required  skills 


•  Interoperability 

•  project  management 
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Abstract 


This  report  analyzes  a  survey  that  the  Software  Engineering  Institute  conducted  on  behalf  of  the  Army 
Strategic  Software  Improvement  Program  (ASSIP).  The  survey  was  directed  to  Army  program 
managers  (PMs)  and  covered  four  areas  of  the  acquisition  system:  the  acquirer’s  environment,  the 
developer’s  environment,  communication  between  the  acquirer  and  developer,  and  external  factors 
that  could  affect  the  acquisition  system.  The  study  aimed  to  discover  how  PMs  perceived  major 
acquisition-related  problem  areas  and  to  provide  preliminary  data  upon  which  to  base  future  data- 
gathering  activities.  Although  the  survey  results  were  not  conclusive,  they  indicated  that  requirements 
management  was  a  primary  area  of  concern  among  those  who  responded  to  the  survey. 
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1  Document  Overview 


Introduction  This  section  discusses  how  the  document  is  organized. 


Document  format  The  document  was  designed  to  support  a  broad  audience  of  readers  with 
varying  degrees  of  information  need.  The  intent  is  to  chunk  and  label 
information  to  support  scanning  (for  example,  for  those  who  are  familiar  with 
aspects  of  the  ASSIP),  while  also  providing  accessible  detail  to  those  who 
have  a  more  comprehensive  need  for  information. 

A  number  of  sections  contain  overviews.  The  purpose  of  the  overview  is  to 
provide  information  that  is  relevant  for  the  subsections  covered  by  the 
overview.  The  overview  also  acts  as  an  information  organizer  to  provide 
readers  with  advance  notice  of  what  to  expect  in  that  section. 


How  this  document  This  table  lists  the  major  sections  of  this  document  and  describes  the  content 
is  organized  of  each. 


Section  title 

Description 

Executive  summary 

An  abstract  of  this  document. 

Overview  of  ASSIP^ 

ASSIP  is  the  improvement  program  for  which  this 
survey  was  conducted. 

ASSIP  data-gathering 
approach 

The  survey  is  only  one  aspect  of  the  overall  ASSIP 
data-gathering  approach.  This  section  describes 
the  overall  approach  and  how  this  survey  is  part  of 
it. 

About  the  survey 

Describes  the  objectives  and  scope  for  this  survey. 

Survey  method 

Describes  the  method  used  for  designing  the 
survey  instrument  and  distributing  it. 

Description  of  the 
survey  sample 

Graphical  charts  describing  various  characteristics 
of  the  surveyed  audience. 

Survey  results 

Reports  the  results  in  chart  format  supplemented  by 
tabular  information. 

Observations  and 
interpretations 

Provides  interpretation  guidelines.  Discusses  the 
results  and  highlights  noteworthy  outcomes. 

Major  themes 

Synthesizes  key  observations  into  major  themes. 

Summary 

A  synopsis  of  the  report  including  next  steps. 

Appendix 

The  questionnaire  instrument  used  in  this  survey. 

^  ASSIP  is  Army  Strategic  Software  Improvement  Program. 
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2  Overview  of  ASSIP 


Introduction 


What  is  ASSIP? 


Established  by 


Purpose 


ASSIP  working 
groups 


Improvement 

vehicle 


The  survey  work  was  conducted  on  behalf  of  the  ASSIP  initiative.  This 
section  provides  background  and  information  about  ASSIP  to  provide  context 
for  the  survey  effort. 


ASSIP  stands  for  Army  Strategic  Software  Improvement  Program. 

The  ASSIP  is  a  long-term  effort  focusing  on  acquisition  programs,  people, 
production  and  sustainment,  and  the  institutionalization  of  continuous 
improvement. 


The  ASSIP  was  established  by  the  Assistant  Secretary  of  the  Army  for 
Acquisition,  Logistics  and  Technology  [ASA(ALT)]. 


The  purpose  of  ASSIP  is  to  take  actions  to  improve  the  acquisition  of  Army 
software-intensive  systems  acquired  for  soldiers  in  pursuit  of  the  Army’s 
Objective  Force. 


This  table  lists  and  describes  the  key  working  bodies  for  guiding  the 
implementation  of  ASSIP. 


Group 

Description 

Senior  Steering  Group 
(SSG) 

ASA(ALT)  Military  Deputy  (MILDEP),  Program 
Executive  Officers  (PEOs)  and  the  SEI  Director 
exercise  management  and  controi  over  ASSIP. 

ASSIP  Action  Group 
(AAG) 

Approximately  1 5  representatives  of  PEOs  who  act 
as  the  development  arm  of  SSG. 

ASSIP  Team 

SEI  team  that  acts  as  Secretariat  supporting  the 

SSG  and  AAG. 

The  Strategic  Software  Improvement  Master  Plan  (SSIMP)  is  the  mechanism 
by  which  improvement  initiatives  are  introduced  into  the  Army  acquisition 
system.  The  SSIMP 

•  is  the  annual  document  that  describes  the  approach  for 
implementation  and  insitutionalization  of  the  ASSIP  during  a  specific 
fiscal  year; 

•  contains  a  suite  of  actionable  improvement  initiatives; 

•  is  revised  on  an  annual  basis  in  response  to  data  (e.g.,  surveys  and 
depth  interviews),  performance-based  measures,  and  lessons 
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learned  from  implementing  the  plan  during  the  previous  year. 


Purpose  of  this  Informed  decision  making  requires  adequate  information  about  the  situation, 

survey  The  survey  (whose  results  are  reported  herein)  is  one  of  several  information¬ 

gathering  approaches  to  support  effective  decision  making  within  the  ASSIP. 
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3  ASSIP  Data-Gathering  Approach 


Introduction 


Data 

triangulation 


ASSIP  use  of 
triangulation 


In-depth 

Interviews 


Surveys 


The  intent  of  ASSIP  is  to  emphasize  a  data  collection  and  analysis  strategy 
that  promotes  the  use  of  objective,  valid  information  to  guide  the  identification 
and  prioritization  of  improvement  opportunities  in  the  Army  acquisition 
environment. 

This  section  provides  a  brief  overview  of  the  data-gathering  approach. 

Data  triangulation  refers  to  the  use  of  multiple  data  collection  and  analysis 
methods  whereby  the  strengths  of  one  method  compensate  for  the 
weaknesses  of  another. 

The  purpose  of  triangulation  is  to  obtain  confirmation  of  findings  through 
convergence  of  different  perspectives.  The  point  at  which  the  perspectives 
converge  is  seen  to  represent  reality. 

Getting  to  the  truth  about  a  system  or  a 
complex  situation  is  difficult. 

Our  survey  method  is  but  one  of  several 
data-gathering  approaches  for 
understanding  the  state  of  the  Army 
acquisition  system.  Other  approaches 
include: 

Interviews  Research 

•  in-depth  interviews  at  selected 
Army  acquisition  sites 

•  consultations  with  members  of  the  ASSIP  Action  Group  (AAG) 

•  interviews  with  acquisition  subject  matter  experts 

•  research  to  explore  the  results  of  other  acquisition-based  data  analysis 
efforts,  assessments,  and  relevant  reports  on  the  state  of  the  practice 

The  ASSIP  Team  is  conducting  a  number  of  in-depth  interviews  at  Army 
acquisition  sites.  These  are  being  referred  to  as  “Program-Wide 
Benchmarking  for  Improvemenf  events. 

The  interviews  are  designed  as  open-ended  exchanges  with  knowledgeable 
individuals  from  the  particular  acquisition  program. 

The  goal  of  these  activities  is  to  enable  the  organization  to  come  to  a  more 
common  understanding  of  its  improvement  issues,  opportunities,  and  current 
best  practices. 

The  results  reported  in  this  document  represent  the  initial  examination  of  the 
Army  acquisition  system.  The  plan  is  to  conduct  an  annual  survey  to  obtain  a 
snapshot  that  portrays  key  aspects  of  the  Army  acquisition  system. 

Information  derived  from  these  surveys  will  be  triangulated  with  other  sources 
(e.g.,  depth  interviews  and  other  valid  sources  of  information). 
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4  About  the  Survey 


Introduction 


Objectives 


Scope  of 
survey 


The  2003  survey  of  Army  acquisition  program  managers  was  the  initial 
ASSIP  data-gathering  event. 


The  objectives  of  this  initial  survey  were  to 

•  provide  preliminary  insight  into  the  major  acquisition-related  problem 
areas  (as  perceived  by  Army  program  managers)  so  that 
improvement  opportunities  can  be  prioritized 

•  assist  development  of  future  data  collection  activities  by  shedding 
light  on  the  problem  space,  thereby  allowing  more  detailed  data- 
gathering  to  be  conducted  (in  the  future)  in  the  identified  focus  areas 


Typically,  the  research  area  for  a  survey  is  narrowly  scoped  to  take  into 
account  the  limitations  of  using  such  a  method.  For  example,  response  rates 
can  be  drastically  affected  by  the  size  of  the  questionnaire.  Therefore,  one 
must  keep  the  size  and  time-to-complete  within  reason. 

When  a  broad  topic  is  selected  as  the  survey  focus,  then  obtaining  adequate 
coverage  of  issues  implies  that  the  information  obtained  is  rather  superficial. 

In  this  survey,  the  researchers  were  requested  to  cover  four  rather  broad 
areas  associated  with  understanding  the  acquisition  system.  These  are 
diagramed  below  and  include 

1.  the  acquirer’s  environment 

2.  communication  between  the  acquirer  and  developer 

3.  the  developer’s  environment 

4.  factors  that  are  external  to  the  acquisition  system  (e.g., 
congressional  mandates,  personnel  turnover  due  to  military  leave) 
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5  Survey  Method 


Introduction 


Survey  content 
resources 


How  the  content 
was  defined 


Survey  format 


Distribution  of 
survey 


This  section  describes  the  process  used  for  the  design  and  distribution  of  the 
survey. 

A  number  of  resources  were  investigated  as  potential  sources  of  content 
material  for  this  survey.  Those  that  had  a  primary  impact  on  defining  the 
content  of  this  survey  included  the  following: 

•  Software  Acquisition  Capability  Maturity  Model  (SA-CMM) 

•  Literature  search  on  acquisition  best  practices 

•  Previous  acquisition-based  surveys,  assessments,  etc. 

•  Subject  matter  expertise  at  the  SEI 

•  AAG  questionnaire  results  (i.e.,  top  5  problems,  issues,  risks) 

•  AAG  input  and  feedback 

The  major  steps  of  survey  content  definition  included  the  following 

1 .  Define  and  validate  survey  objectives  with  team  and  stakeholders. 

2.  Draft  questions. 

3.  Review  and  obtain  feedback  from  ASSIP  Team. 

4.  Revise  questions  and  develop  draft  survey  instrument. 

5.  Present  instrument  to  AAG  and  customer  point  of  contact  for  review 
and  feedback. 

6.  Review  AAG  feedback  and  revise  survey  instrument. 

7.  Conduct  final  review  of  survey  instrument  with  ASSIP  Team. 

8.  Conduct  final  edit  of  survey  instrument. 


The  survey  was  a  structured,  self-administered  questionnaire  that  was 
available  both  electronically  via  the  World  Wide  Web  and  in  paper  form. 

The  questions  were  phrased  in  pre-coded  “close-ended”  format.  In  addition, 
there  were  two  “open-ended”  questions  so  that  respondents  could  elaborate 
their  opinions  and  suggestions  for  improvement. 


The  survey  was  a  Web-based  instrument  that  was  available  via  the  SEI’s 
external  server  site. 

The  AAG  Chair  emailed  members  of  the  AAG  requesting  that  they  contact 
program  managers  within  their  organization  to  complete  the  Web-based 
survey. 

The  response  window  for  the  survey  was  originally  set  for  two  weeks.  As  the 
deadline  approached,  management  decided  to  extend  the  deadline  for  an 
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additional  three  and  a  half  weeks. 

A  number  of  reminder  emails  (from  both  the  AAG  Co-Chair  and  from  the 
Secretariat)  were  sent  to  AAG  members  to  encourage  100  percent 
participation  of  their  managers. 
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Description  of  the  Survey  Sampie 


introduction 


Target 

audience 


Not  a  random 
sample 


Sampie 

contamination 


Characteristics  of  the  survey  sampie  are  presented  in  this  section.  Areas  of 
interest  included  information  that  profiled  both  the  respondent  and  the 
respondent’s  organization. 


The  target  audience  for  this  survey  was  Army  acquisition  program  managers. 
There  is  variation  to  how  this  term  is  appiied  and  so  we  inciuded  as  part  of 
the  intended  survey  audience  individuals  who  identified  themselves  as 

•  Program  Managers 

•  Project  Managers 

•  Product  Managers 

•  Project  Directors 


To  enable  generalizations  to  be  made  from  any  survey,  the  researcher  must 
be  very  careful  to  ensure  that  an  appropriate  random  sampie  is  obtained  from 
the  population  under  study.  When  a  random  sample  is  generated,  the 
researcher  can  then  be  confident  that  the  sample  actually  represents  the 
population  of  interest. 

In  this  study,  it  was  not  possible  to  obtain  a  random  sample.  Program 
managers  (from  the  Army  acquisition  community)  were  requested  to 
participate  in  this  survey,  and  so  respondents  were  self-seiected.  We  had 
limited  insight  into  the  home  organization  of  participants  who  responded  to 
this  survey  anonymously. 

Since  an  appropriate  random  sample  was  not  obtained,  the  results  of  this 
survey  cannot  be  generalized  to  the  overall  population  of  Army  acquisition 
program  managers  (who  are  responsible  for  software-intensive  systems). 


Most  of  the  respondents  were  program  managers.  However,  in  some  cases, 
it  appears  that  the  task  of  responding  to  this  survey  was  delegated  to  other 
individuals  within  the  organization  including 

•  Software  Engineers  (14,  9%) 

•  Contractors  (6, 4%) 

•  Team  Leaders  (6, 4%) 
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Figure  1 :  Proportion  of  Army  acquisition  managers  who  responded  to  this  survey. 


Population  of  Army 


Program  managers  who  did 
participate  in  survey  (150) 


Respondents  Based  on  information  obtained  from  the  RDAISA*  Database,  there  is  a  total  of 

approximately  500  acquisition  managers  in  the  Army  Acquisition  Workforce. 

Approximately  150  Acquisition  managers  participated  in  this  survey.^ 

Approximately  350  Acquisition  managers  did  not  participate  in  the  survey.  Of 
these,  some  are  not  responsible  for  software-based  acquisition  and  so  were 
not  included  in  the  request  for  participation.  However,  it  is  unknown  what 
proportion  of  the  overall  population  that  this  represents. 
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RDAISA  is  Army  Research  Development  and  Acquisition  Information  Systems  Activity.  The  information  in  this 
database  changes  over  time.  The  most  recent  information  from  this  database  (during  the  final  edit  of  this  report) 
Indicates  that  there  are  actually  576  programs  under  the  control  of  ASA(ALT).  However,  not  all  of  these  programs 
Include  software  as  a  component. 

^  In  some  cases,  it  appears  that  the  survey  was  delegated  by  the  Program  Manager  to  someone  else  in  the 
organization  to  complete.  (See  Figure  3.) 
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What  organizations 
are  respondents 
from? 


In  general,  we  know  little  about  the  home  organization  of  the  respondents. 
During  and  after  the  time  that  the  survey  was  accessible  on  the  Internet , 
respondents  were  requested  to  notify  their  ASSIP  Action  Group  (AAG) 
representative  to  inform  them  as  to  whether  they  had  completed  the  survey. 

Fifty  of  the  respondents  did  report  that  they  had  completed  the  survey.  This 
table  lists  the  organization  and  the  number  of  respondents  from  each 
organization  who  reported  that  they  did  complete  the  survey. 

This  information  was  self-reported  and  there  is  no  way  for  us  to 
independently  validate  this  information. 


Organization 

PEG  STRI 
PEG  iEWS 
PEG  AMMG 
JTRS  JPG 
USA  PEG  C3T 
Unknown 


#  Respondents 
4 
20 
6 
1 

19 

100 
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Response 


Respondent  Respondents  were  asked  to  indicate  the  role  that  best  describes  their 

Role  position  within  the  organization.  Figure  2  displays  the  results. 

A  majority  of  the  respondents  identified  themselves  as  Project  or  Product 
Managers  (48%). 

Figure  2:  Role  of  respondent  within  organization 


Project  or  Product 
Manager 


Deputy  Program 
Manager 


Program  Manager 


Program  Executive 
Officer  (PEO) 


Other  (please  describe) 


No  Response 


0  10  20 


30  40  50  60  70 


80 


Frequency 


“Other”  category  Thirty-four  percent  of  the  respondents  chose  “Other”  from  the  provided 

responses.  Figure  3  (on  the  following  page)  shows  a  summary  of  the  role  that 
this  category  described. 
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Response 


Figure  3;  Role  description  for  “Other”  category  (of  Figure  2) 


Engineer 


Director  or  Chief 


Deputy  or  Assistant 
Product  Manager 


Consultant  or  Contractor 


Project  and  Team 
Leaders 

Assistant  Project 
Manager  or  Manager 

0  10  20 

Frequency 


Contamination  of  Instructions  for  this  survey  stated  that  the  respondent  should  be  a  project  or 
sample  product  manager  within  the  organization.  Yet  it  appears  that  the  sample  was 

contaminated  with  individuals  who  were  not  intended  as  part  of  the  sample. 

One  could  speculate  that  the  request  (for  participation)  was  delegated  by  the 
project  manager  to  someone  who  was  perceived  to  be  more  qualified  to 
respond  to  the  questions  (that  focus  on  the  software  aspect  of  systems).  One 
could  also  speculate  that  the  respective  project  manager  responded  to  the 
survey  and  also  requested  staff  members  (within  their  group)  to  respond  as 
well.  However,  the  actual  circumstances  are  unknown  to  the  researcher. 
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Response 


Figure  4 


Rank  or  Job  Classification  of  Respondent 
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Figure  5:  Years  of  experience  working  as  an  Army  acquisition  manager 
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Response 


Figure  7:  Self-reported  level  of  expertise  in  acquisition  program/project  management 
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Figure  8:  Self-reported  level  of  expertise  in  software  acquisition  management 
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Response 


Figure  10:  Self-reported  level  of  expertise  In  systems  engineering 
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Explanation  of 
diagram 


Type  of  systems 
being  acquired 


Figure  1 1 :  Respondent’s  responsibility  for  ACAT  Programs 


H  =  224 


The  150  individuals  that  responded  to  this  survey  identified  the  ACAT  level  of 
the  programs  that  they  are  responsible  for. 

Since  the  total  number  of  programs  identified  is  224,  it  is  obvious  that  some 
managers  are  responsible  for  multiple  programs  (and  these  are  a  mixture  of 
ACAT  levels). 


Respondents  were  asked  to  describe  the  type  of  system(s)  and  how  many  of 
each  type  that  they  are  responsible  for.  This  table  summarizes  the  figures 
that  are  presented  on  the  upcoming  pages. 


Fig. 

Type  of 
system 

Description 

12 

Automated 

Information 

Systems 

Management  information  systems  supporting  business 
operations  such  as  payroll,  inventory,  or  logistics 

13 

Weapons 

Systems 

Systems  with  real-time  process  control  or  guidance 
systems  for  avionics  or  radar;  embedded  software 
running  in  electronic  devices,  vehicies,  missiles  or 
aircraft 

14 

C^IEW  or 
C4ISR 

Decision  support  systems,  intelligence  systems,  mission 
planning,  communications  systems,  or  maneuver  control 

15 

Other 

Modeling  and  simulation,  compilers,  configuration 
management  tools,  cost  estimation  toois,  personal 
computer  applications,  pattern  recognition 

BEST  AVAILABLE  COPY 


CMU/SEI-2004-TR-003 


23 


>5 


3-5 


2 


1 
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No  Response 


Figure  12:  Automated  Information  Systems  per  Respondent 
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Total  number 

150 
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1% 

3-5 
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9% 

0 

61% 
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22% 

The  term  “automated  information  systems”  refers  to  management  information 
systems  supporting  business  operations  such  as  payroll,  inventory,  or 
logistics. 
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Figure  13:  Number  of  Weapons  Systems  per  Respondent 
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Total  number 

150 

>5 

5% 

3-5 

11% 

2 

11% 

1 

21% 

0 

39% 

No  Response 

13% 

Note  The  term  “weapons  systems”  refers  to  systems  with  real-time  process  control 

or  guidance  systems  for  avionics  or  radar;  embedded  software  running  in 
electronic  devices,  vehicles,  missiles  or  aircraft. 
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Figure  14:  Number  of  C  lEW  or  C4ISR  per  Respondent 
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The  term  “C^IEW  or  C4ISR"  refers  to  decision  support  systems,  intelligence 
systems,  mission  planning,  communications  systems,  or  maneuver  control. 
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Figure  15:  Number  of  Other  Systems  per  Respondent 
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Note  The  term  “Other”  refers  to  such  systems  as  modeling  and  simulation, 

compilers,  configuration  management  tools,  cost  estimation  tools,  personal 
computer  applications,  pattern  recognition. 
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7  Overview  of  Survey  Results 


Introduction 


Result  categories 


In  this  chapter,  the  survey  results  are  presented  in  graphical  format 
supported  by  tabular  information. 

The  results  are  not  commented  on  in  this  chapter.  For  key  observations  and 
interpretations,  see  chapter  8,  Observations  and  Conclusions  on  page  103. 


This  chapter  contains  the  following  categories  of  results. 


Results 

See  Page 

The  acquirer  environment  and  communication 

32 

The  developer  environment 

84 

Impact  of  external  factors  on  acquisition 

112 

Where  are  the  major  problems  and  risks? 

113 

What  are  the  most  difficult  problems? 

115 
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7.1  Relevancy  of  Current  DoD  Acquisition-Based  Initiatives 

Background  The  DoD  has  implemented  the  major  acquisition-based  initiatives  identified 

below: 

•  Performance-Based  Payments 

•  Price-based  Acquisition 

•  Performance-Based  Services  Acquisition 

•  Interoperability 

•  Cost  as  an  Independent  Variable 

•  Electronic  Business 

•  Knowledge  Management  in  the  Acquisition  Workforce 

•  Past  Performance 

•  Performance  Specifications 

•  FAR  Part  1 2  /  Commercial  Item  Procurements 

•  Total  Ownership  Cost  Management 

•  ISO  9001 : 2000  Quality  Management  Systems 

•  Open  Systems  Design 

•  Integrated  Digital  Environment 

•  Logistics  Transformation 

•  Contractor  Incentives  Guide 

•  Reducing  Government  Property  in  Possession  of  Contractors 

•  Intellectual  Property  Guidelines 

Respondents  were  asked  to  indicate  their  perception  of  the  relevance  of 
these  initiatives  to  how  they  perform  their  work. 

Scaling  the  results  To  summarize  the  results,  we  applied  the  following  numerical  scale  to  the 
available  response  categories: 


Response  category 

Score 

Highly  relevant 

5 

Relevant 

3 

Somewhat  relevant 

1 

Not  relevant  or  don’t  know 

0 

The  response  totals  were  summed  for  each  initiative,  then  divided  by  the 
number  of  respondents  to  normalize  the  assigned  score. 

The  results  appear  on  the  next  page  in  Figure  16. 
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Figure  16:  Respondent  perception  of  relevancy  of  current  DoD  acquisition-based  initiatives 
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7.2  The  Acquirer  Environment  &  Communication  - 
Overview 

Context  One  section  of  the  survey  focused  on  the  acquirer  environment  and  the 

diagram  communication  between  acquirer  and  deveioper  (shaded  areas  in  diagram 

below).  This  section  describes  the  results  for  these  topic  areas. 


Factors  externai  to  Acquisition  System  that  impact  it 


Communication 

. ..... 


The  Acquisition  System 


Acquirer 

•Environment 


Deveioper 

Environment 


General  format  of  In  general,  the  questionnaire  items  for  this  section  were  close-ended  with 
questions  possible  response  categories  as  foliows: 

•  Strongly  Agree 

•  Agree 

•  Disagree 

•  Strongly  Disagree 

•  Don’t  Know  or  Not  Applicable 

In  this  section  Questions  focused  on  the  following  topical  categories  listed  in  this  table. 


Topic  Area 

See  Page 

Education  and  training 

33 

Software  acquisition  planning  and  project  management 

39 

Use  of  measurement  to  support  decision  making 

49 

Solicitation 

57 

Software-related  contractual  requirements  development  and 
management 

63 

Contract  performance  management 

69 

T  ransition-to-support 

74 

Transition-to-operations 

79 

For  each  topic  For  each  topic  area  listed  in  the  table  above,  a  summary  of  the  results  are 

area  first  presented,  followed  by  charts  that  present  additional  detail.  The 

summary  shows  the  combined  “Strongly  Agree”  and  ’’Agree”  tallies  for  each 
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question. 

In  general,  a  response  showing  agreement  suggests  that  an  effective 
practice  is  in  use  by  the  respondent’s  organization. 

7.2.1  Education  and  Training 

Synopsis  This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 

Agree”  or  “Agree”  for  their  responses. 


Questionnaire  Item 

Agree 

Training  that  is  required  for  the  project  teams  to  achieve  their 
software  acquisition  objectives  is  identified  and  provided. 

73 

You  know  who  to  contact  when  you  have  training  needs  from 
the  organization. 

88 

There  are  mechanisms  in  place  that  allow  you  to  provide 
feedback  with  regard  to  the  effectiveness  of  training. 

73 

You  use  organizational  or  project  training  plans  to  plan  for 
individual  training. 

59 

In  general,  you  feel  there  are  ample  training  opportunities 
available  to  ensure  that  project  staff  have  the  right  skills  to 
perform  their  jobs. 

83 

Detailed  response  The  detailed  response  profiles  are  presented  on  the  following  pages. 

profiles 


CMU/SEI-2004-TR-003 


33 


Response 


Figure  17:  Training  that  is  required  for  the  project  teams  to  achieve  their  software  acquisition 
objectives  is  identified  and  provided. 
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Figure  18:  You  know  who  to  contact  when  you  have  training  needs  from  the  organization 
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Response 


Figure  19:  There  are  mechanisms  in  place  that  allow  you  to  provide  feedback  with  regard  to  the 
effectiveness  of  training. 
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Figure  20:  You  use  organizational  or  project  training  plans  to  plan  for  individual  training. 
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Response 


Figure  21 :  In  general,  you  feel  there  are  ample  training  opportunities  available  to  ensure  that 
project  staff  have  the  right  skills  to  perform  their  jobs. 
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7.2.2  Software  Acquisition  Pianning  and  Project  Management 


Synopsis 


Detailed  response 
profiles 


This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 
Agree”  or  “Agree”  for  their  responses. 


Questionnaire  Item 

Agree 

Software  experts  participate  in  system  acquisition  planning. 

84 

Acquisition  plans  are  revised  when  major  changes  occur. 

87 

Project-wide  participation  in  the  identification  and  mitigation 
of  risks  is  encouraged  and  valued  by  management. 

85 

Your  acquisition  project  assesses  the  likeiihood  and 
consequence  of  each  risk  and  monitors  the  status  of  each 
risk  item. 

85 

A  novice  engineer  (participating  on  this  acquisition  project) 
would  know  how  to  surface  risks  according  to  the  risk 
identification  and  analysis  plan. 

59 

Weekly  or  biweekly  status  checks  or  other  periodic  reviews 
are  held  to  manage  and  control  risks,  issues,  and  problems 
discovered  during  the  software  acquisition. 

81 

If  a  novice  engineer  discovers  a  problem  or  risk  in  the 
system  design,  1  am  confident  that  they  would  know  what  to 
do  to  surface  that  issue. 

75 

There  is  a  well-understood  and  effective  process  for 
resolving  issues  among  all  project  functions. 

76 

There  is  a  change  request  process  for  submitting 
suggestions  for  improving  the  acquisition  process. 

56 

The  detailed  response  profiles  are  presented  on  the  following  pages. 
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Response 


Figure  22:  Software  experts  participate  in  system  acquisition  planning. 
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Figure  23:  Acquisition  plans  are  revised  when  major  changes  occur. 
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Response 


Figure  24:  Project-wide  participation  in  the  identification  and  mitigation  of  risks  is  encouraged 
and  valued  by  management. 
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Figure  25:  Your  acquisition  project  assesses  the  likelihood  and  consequence  of  each  risk  and 
monitors  the  status  of  each  risk  item. 
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Response 


Figure  26:  A  novice  engineer  (participating  on  this  acquisition  project)  would  know  how  to 
surface  risks  according  to  the  risk  identification  and  analysis  plan. 
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Figure  27:  Weekly  or  biweekly  status  checks  or  other  periodic  reviews  are  held  to  manage  and 
control  risks,  issues,  and  problems  discovered  during  the  software  acquisition. 
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Response 


Figure  28:  If  a  novice  engineer  discovers  a  problem  or  risk  in  the  system  design,  I  am  confident 
that  they  would  know  what  to  do  to  surface  that  issue. 
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Response 


Figure  29:  There  is  a  well-understood  and  effective  process  for  resoiving  issues  among  all 
project  functions. 
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7.2.3  Use  of  Measurement  to  Support  Decision  Making 


Synopsis 


Metrics 

used 


Detaiied  response 
profiles 


This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 
Agree”  or  “Agree”  for  their  responses. 


Questionnaire  Item 

Agree 

Planning  estimates  are  based  on  historical  measurement 
data  from  previous  acquisition  projects. 

83 

Measurement-based  objectives  for  the  acquired  products 
and  services  are  defined. 

79 

The  acquisition  project  uses  metrics  as  an  input  to  program 
decision  making. 

81 

The  performance,  cost,  and  schedule  objectives  of  the 
software  acquisition  project  are  measured  and  controlled 
throughout  the  software  acquisition 

75 

Your  project  team  uses  measures  and  analytic  techniques 
for  statistically  managing  selected  processes  and  sub¬ 
processes. 

67 

Your  project  team  records  statistical  and  quality 
management  data  in  the  organization’s  measurement 
repository  and  uses  that  information  for  decision  making. 

50 

Respondents  were  also  asked  to  identify  the  types  of  measures  used  to  track 
project  status.  Below  is  a  summary  of  these  results.  A  detailed  chart  of  this 
information  is  provided  in  Figure  37on  page  56. 


Measure 

using 

Requirements  stability 

51 

Manpower 

63 

Development  progress 

82 

Cost 

86 

Schedule 

91 

The  detailed  response  profiles  are  presented  on  the  following  pages. 


CMU/SEI-2004-TR-003 


49 


Response 


Figure  31 :  Planning  estimates  are  based  on  historical  measurement  data  from  previous 
acquisition  projects. 
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Figure  32:  Measurement-based  objectives  for  the  acquired  products  and  services  are  defined. 
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Figure  34:  The  performance,  cost,  and  schedule  objectives  of  the  software  acquisition  project  are 
measured  and  controlled  throughout  the  software  acquisition. 
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Figure  35:  Your  project  team  uses  measures  and  analytic  techniques  for  statistically  managing 
selected  processes  and  sub-processes. 
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Figure  36:  Your  project  team  records  statistical  and  quality  management  data  in  the 

organization’s  measurement  repository  and  uses  that  information  for  decision  making. 
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Figure  37:  The  following  metrics  are  reported  to  the  PMO  on  (at  least)  a  monthly  basis. 


Requirements  stability 


Manpower 


Development  progress 


Cost 


Schedule 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


LEGEND 

E  Yes 
■  No 


Percent 


Total  number  of  respondents  was  150 


BEST  AVAILABLE  COPY 


6 


CMU/SEI-2004-TR-003 


7.2.4  Solicitation 


Synopsis  This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 

Agree”  or  “Agree”  for  their  responses. 


Questionnaire  Item 

Agree 

The  selection  official  has  sufficient  software  technical 
expertise  to  select  a  qualified  contractor. 

64 

The  software-related  contractual  requirements  baseline  is 
established  prior  to  release  of  the  solicitation  package. 

63 

The  solicitation  package  includes  the  contractual  software 
requirements  and  proposal  evaluation  criteria. 

72 

Technical  reviewers  use  proposal  evaluation  criteria  during 
solicitation  activities. 

80 

Software  risks  are  independently  evaluated  as  part  of  the 
solicitation  and  are  communicated  to  the  solicitation  official. 

58 

Detailed  response  The  detailed  response  profiles  are  presented  on  the  following  pages. 

profiles 
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Figure  39:  The  software-related  contractual  requirements  baseline  is  established  prior  to  release 
of  the  solicitation  package. 
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Response 


Figure  40:  The  solicitation  package  includes  the  contractual  software  requirements  and  proposal 
evaluation  criteria. 
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Figure  41 :  Technical  reviewers  use  proposal  evaluation  criteria  during  solicitation  activities 
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Figure  42:  Software  risks  are  independently  evaluated  as  part  of  the  solicitation  and  are 
communicated  to  the  solicitation  official. 
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7.2.5  Software-Related  Contractual  Requirements  Development  and  Management 


Synopsis  This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 

Agree”  or  “Agree”  for  their  responses. 


Questionnaire  Item 

Agree 

Software-related  contractual  requirements  are  developed, 
managed,  and  maintained  using  a  structured  process. 

79 

End  users  and  other  affected  groups  have  input  to  the 
software-related  contractual  requirements  over  the  life  of  the 
acquisition. 

78 

A  member  of  the  acquisition  project  staff  or  a  novice 
engineer  could  identify  and  verify  the  source  of  software- 
related  contractual  requirements. 

66 

In  the  case  of  new  and/or  changing  program  requirements, 
acquisition  project  staff  know  when  and  how  to  make 
changes  to  contractual  requirements,  including  acceptance 
criteria. 

79 

A  formal  control  board  is  in  place  to  authorize  changes  to 
requirements. 

73 

Detailed  response  The  detailed  response  profiles  are  presented  on  the  following  pages. 

profiles 
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Figure  43:  Software-related  contractual  requirements  are  developed,  managed,  and  maintained 
using  a  structured  process. 
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Figure  44:  End  users  and  other  affected  groups  have  input  to  the  software-related  contractual 
requirements  over  the  life  of  the  acquisition. 
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Figure  45:  A  member  of  the  acquisition  project  staff  or  a  novice  engineer  couid  identify  and  verify 
the  source  of  software-reiated  contractuai  requirements. 
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Figure  46:  In  the  case  of  new  and/or  changing  program  requirements,  acquisition  project  staff 
know  when  and  how  to  make  changes  to  contractual  requirements,  including 
acceptance  criteria. 
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Figure  47:  A  formal  control  board  is  in  place  to  authorize  changes  to  requirements. 
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7.2.6  Contract  Performance  Management 


Purpose 


Synopsis 


Detailed  response 
profiles 


The  purpose  of  the  set  of  questionnaire  items  presented  in  this  section  was  to 
address  whether  the  respondent’s  project  team  ensures  that  the  software 
activities  under  contract  are  being  performed  in  accordance  with  contractual 
requirements. 


This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 
Agree”  or  “Agree”  for  their  responses. 


Questionnaire  Item 

Agree 

The  project  team  has  sufficient  insight  into  the  contractor’s 
software  engineering  effort  to  ensure  that  the  effort  is 
managed  and  controlled  and  complies  with  contract 
requirements. 

78 

The  acquisition  project  team  and  contractor  team  maintain 
ongoing  communication  and  both  parties  agree  to 
commitments. 

86 

Your  project  team  identifies,  documents,  and  tracks  software 
risks  associated  with  the  contractor’s  efforts,  independent  of 
the  contractor’s  risk  management  process. 

67 

The  quality  of  the  contractor  team’s  process,  performance, 
products,  and  services  are  appraised  throughout  the 
contract’s  period  of  performance  to  identify  risks  and  take 
action  to  mitigate  those  risks  as  early  as  possible. 

78 

The  detailed  response  profiles  are  presented  on  the  following  pages. 
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Figure  48:  The  project  team  has  sufficient  insight  into  the  contractor’s  software  engineering  effort 
to  ensure  that  the  effort  is  managed  and  controiled  and  compiies  with  contract 
requirements. 
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Figure  49:  The  acquisition  project  team  and  contractor  team  maintain  ongoing  communication 
and  both  parties  agree  to  commitments. 
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Response 


Figure  50:  Your  project  team  identifies,  documents,  and  tracks  software  risks  associated  with  the 
contractor’s  efforts,  independent  of  the  contractor’s  risk  management  process. 
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Figure  51 :  The  quality  of  the  contractor  team’s  process,  performance,  products,  and  services  are 
appraised  throughout  the  contract’s  period  of  performance  to  identify  risks  and  take 
action  to  mitigate  those  risks  as  early  as  possible. 


Strongly  Agree 


Agree 


Disagree 


Strongly  Disagree 


Don’t  Know  or  N/A 


No  Response 


0  10  20  30  40  50  60  70  80 

Frequency 


i  Total  number 

150 

Strongly  Agree  and  Agree 

78% 

Strongly  Disagree  and  Disagree 

11% 

Don’t  Know  or  N/A 

8% 

CMU/SEI-2004-TR-003 


73 


7.2.7  Transition  to  Support 


Purpose  The  purpose  of  the  set  of  questionnaire  items  presented  in  this  section  was  to 

address  whether  projects  provide  for  the  transition  of  the  software  products 
being  acquired  to  the  eventual  software  support  organization. 


Synopsis  This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 

Agree”  or  “Agree”  for  their  responses. 


Questionnaire  Item 

Agree 

The  acquisition  project  team  ensures  that  the  software 
support  organization  has  the  capacity  and  capability  to 
provide  the  required  support  upon  assumption  of 
responsibility  for  the  support  of  the  software  products. 

70 

The  acquisition  project  team  ensures  that  there  is  no  loss  in 
continuity  of  support  to  the  software  products  during 
transition  from  the  development  contractor  to  the  software 
support  organization. 

69 

Configuration  management  of  the  software  products  is 
maintained  throughout  the  transition. 

70 

The  strategy  for  transition  into  maintenance  is  documented, 
communicated,  and  agreed  to  by  all  parties  early  in  the 
acquisition. 

66 

Detailed  response  The  detailed  response  profiles  are  presented  on  the  following  pages. 

profiles 
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Figure  52:  The  acquisition  project  team  ensures  that  the  software  support  organization  has  the 
capacity  and  capability  to  provide  the  required  support  upon  assumption  of 
responsibility  for  the  support  of  the  software  products. 
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Response 


Figure  53:  The  acquisition  project  team  ensures  that  there  is  no  loss  in  continuity  of  support  to 
the  software  products  during  transition  from  the  development  contractor  to  the 
.  software  support  organization. 
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Figure  54:  Configuration  management  of  the  software  products  is  maintained  throughout  the 
transition. 
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Figure  55:  The  strategy  for  transition  into  maintenance  is  documented,  communicated,  and 
agreed  to  by  ali  parties  early  in  the  acquisition. 
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7.2.8  Transition  to  Operations 


Purpose 


Synopsis 


Detailed  response 
profiles 


The  purpose  of  the  set  of  questionnaire  items  presented  in  this  section  was  to 
address  whether  acquisition  projects  prepare  for  the  transition  of  the  software 
products  being  acquired  to  the  end  user. 


This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 
Agree”  or  “Agree”  for  their  responses. 


Questionnaire  Item 

Agree 

The  acquisition  project  team  ensures  that  the  end  user  has 
the  training,  experience,  and  resources  to  accept  the 
software  products  into  operational  use. 

83 

The  acquisition  project  team  plans  for  sufficient  contractor 
support  during  end-user  acceptance  testing. 

85 

The  strategy  for  transition  into  operations  is  documented, 
communicated,  and  agreed  to  by  all  parties  in  the 
acquisition. 

81 

The  software  support  organization  participates  in  all  project 
and  technical  reviews. 

70 

The  detailed  response  profiles  are  presented  on  the  following  pages. 
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Figure  56:  The  acquisition  project  team  ensures  that  the  end  user  has  the  training,  experience, 
and  resources  to  accept  the  software  products  into  operational  use. 
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Figure  57:  The  acquisition  project  team  plans  for  sufficient  contractor  support  during  end-user 
acceptance  testing. 
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Figure  58:  The  strategy  for  transition  into  operations  is  documented,  communicated,  and  agreed 
to  by  ali  parties  in  the  acquisition. 
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Figure  59:  The  software  support  organization  participates  in  all  project  and  technical  reviews. 
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7.3  The  Developer  Environment 

Context  In  this  section,  we  describe  the  results  of  survey  items  that  focused  on 

diagram  questions  about  the  “Developer  Environment”  (shaded  areas  in  diagram 

below). 


Factors  external  to  Acquisition  System  that  impact  it 


Acquirer 

Environment 


o 


Communication 


Developer 

"Environment 


The  Acquisition  System 


Explanation 


Categories  that 
were  rated 


Respondents  were  asked  to  indicate  their  perception  of  the  impact  of  the 
contractor’s  work  processes  on  the  success  of  their  software-intensive 
system  acquisitions. 

To  accomplish  this,  respondents  rated  a  list  of  work  processes  by  assigning 
one  of  the  following  to  refiect  how  they  view  each  of  the  contractor’s 
processes:  (a)  Excellent,  (b)  Above  average,  (c)  Average,  (d)  Below  average, 
(d)  Extremely  Poor,  (f)  Don’t  know. 

The  categories  that  were  rated  by  the  respondents  are: 

•  Project  planning 

•  Project  monitoring  and  control 

•  Requirements  development 

•  Requirements  management 

•  Measurement  analysis  and  reporting 

•  Software  architecture  development  and  assessment 

•  Technical  solution 

•  Product  integration 

•  Configuration  Management 

•  Risk  management 

•  Verification 

•  Validation 

•  Supplier  or  subcontract  management 

•  Integrated  teaming 

•  Defined  processes  that  support  product  development  and  stable 
operations 

•  Sharing  all  relevant  information  that  you  feel  you  should  know  to 
manage  the  acquisition  effectively 

•  Causal  analysis  and  resolution  for  defect  prevention 
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How  the  results  are 
presented 


The  response  categories  were  assigned  a  scaied  numerical  vaiue  as  shown 
in  this  table: 


Response  category 

Numerical 

assignment 

Excellent 

5 

Above  average 

3.5 

Average 

2.5 

Below  average 

1 

Extremely  Poor 

0 

Don’t  know  or  didn’t  respond 

0 

The  responses  were  summed  and  normalized  by  dividing  the  summed 
responses  by  the  number  of  respondents. 

The  results  are  presented  as  Figure  60  on  the  next  page. 
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Figure  60;  Scaled  rating  of  respondent’s  perception  of  impact  of  the  contractor’s  work 
processes  on  the  success  of  their  software-intensive  system  acquisitions 
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7.4  Impact  of  External  Factors  on  Acquisition 


Context 

diagram 


In  this  section,  responses  are  focused  on  questions  about  factors  that  are 
perceived  to  be  external  to  the  acquisition  system  (but  may  impact  it). 

This  is  represented  in  the  diagram  below  by  the  shaded  area. 


I  Factors  external  to  Acquisition  System  that  impact  it 
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Communication^''^ 

Developer 
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II 

1 

The  Acquisition  System 

1 

Synopsis  This  table  shows  the  percentage  of  respondents  who  selected  “Strongly 

Agree”  or  “Agree"”  for  their  responses  to  survey  items. 


Questionnaire  Item 

Agree 

Mandates  from  Congress  inhibit  our  program  from  meeting 
its  goals. 

20 

Reallocation  of  program  funding  is  a  significant  source  of 
frustration  in  acquisition  programs. 

90 

Critical  personnel  are  lost  due  to  military  rotations  or  inability 
to  compete  with  industry  salaries. 

55 

Acquisition  reform  has  negatively  impacted  our  ability  to 
meet  our  objectives. 

20 

Expression  of  user  requirements  throughout  the  acquisition 
process  causes  disruption  in  the  development  process. 

20 

Lack  of  test  bed  assets  to  stress  test  system  under  realistic 
operational  conditions  is  a  major  problem. 

20 

Detailed  response  The  detailed  response  profiles  are  presented  on  the  following  pages. 

profiles 
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Response 


Figure  61:  Mandates  from  Congress  inhibit  our  program  from  meeting  its  goals. 
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Figure  62:  Reallocation  of  program  funding  is  a  significant  source  of  frustration  in  acquisition 
programs. 
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Figure  63:  Critical  personnei  are  lost  due  to  military  rotations  or  inability  to  compete  with  industry 
salaries. 
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Figure  64:  Acquisition  reform  has  negativeiy  impacted  our  ability  to  meet  our  objectives. 
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Figure  65:  Expression  of  user  requirements  throughout  the  acquisition  process  causes 
disruption  in  the  development  process. 
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Figure  66:  Lack  of  test  bed  assets  to  stress  test  system  under  realistic  operational  conditions  is  a 
major  problem. 
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7.5  Where  Are  the  Major  Problems  and  Risks? 


Explanation 


Respondents  were  asked  to  identify  where  they  perceived  the  major  problems  and 
risks  in  their  acquisition  programs  to  be.  The  areas  considered  were  listed  as 

1 .  acquisition  program  policies  and  processes 

2.  the  contracting  process  between  acquirers  and  developers 


3.  the  contractor’s  (or  supplier’s)  development  process 

4.  factors  outside  the  control  of  the  acquirers  and  developers  (e.g., 
congressional  mandates,  priorities  set  by  engagements  of  our  armed 
forces) 


|Factore  external  to  Acquisition  System  that  impact  it  ■  <  ’  ' 
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As  an  additional  guideline  for  responding  to  this  item,  participants  were  asked  to 
consider  their  cumulative  experience  in  acquisition  over  the  last  1 0  years. 


Figure  67  presents  a  summary  of  the  results.  To  portray  this  summary,  the 
following  numerical  scale  was  assigned  to  response  categories. 


To  a  great  extent  =5  Average  =1 

Somewhat  =  3  Not  at  all  =  0 


The  response  totals  were  summed  for  each  of  the  four  domain  areas  and  then 
divided  by  the  number  of  respondents  to  normalize  the  assigned  score. 
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Figure  67:  Weighted  responses  that  indicate  where  respondents  believe  the  majority  of  problems 
and  risks  (affecting  their  acquisition  projects)  reside 
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Explanation  For  displaying  the  resuits  above,  the  response  totals  were  summed  for  each 

of  the  four  domain  areas,  then  divided  by  the  number  of  respondents  to 
normaiize  the  assigned  score. 


Detailed  response  The  detailed  response  profiles  are  presented  on  the  following  pages. 

profiles 
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Response 


7.5.1  Acquisition  Programs  and  Poiicies 

Figure  68;  Respondents’  perception  of  the  extent  to  which  acquisition  policies  and  processes 
contribute  to  the  major  problems  and  risks  of  acquisition  programs 
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7.5.2  Contracting  Pocess  Between  Acquirer  and  Developer 


Figure  69:  Respondents’  perception  of  the  extent  to  which  the  contracting  process  between 
acquirer  and  developer  contributes  to  the  major  problems  and  risks  of  acquisition 
programs 
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7.5.3  Contractor’s  (or  Supplier’s)  Development  Process 


Figure  70:  Respondents’  perception  of  the  extent  to  which  the  contractor’s  (or  supplier’s) 
development  process  contributes  to  the  major  problems  and  risks  of  acquisition 
programs 
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7.5.4  Factors  Outside  the  Control  of  the  Acquirers  and  Developers 


Figure  71 :  Respondents’  perception  of  the  extent  to  which  factors  outside  the  control  of  the 

acquirers  and  developers  (congressional  mandates,  priorities  set  by  engagements  of 
our  armed  forces,  etc.)  contribute  to  the  major  problems  and  risks  of  acquisition 
programs 
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7.6  What  Are  the  Most  Difficult  Problems? 


Explanation 


How  the  results  are 
being  reported 


Analysis  approach 
to  narrative 
responses 


Charting  the 
results 


The  survey  included  two  closely  related,  open-ended  questions: 

Question  1 

Overall,  what  are  the  one  or  two  most  difficult  problems  that  your  project 
has  faced  in  conducting  successful  acquisition  of  software-intensive 
systems?  (Please  describe.) 

Question  2 

If  you  could  change  one  or  two  things  about  how  software-intensive 
systems  are  acquired  in  the  Army,  what  would  they  be?  (Please 
describe.) 


To  preserve  anonymity,  the  narrative  responses  are  not  being  reported  in  this 
document. 

Question  1 

In  this  section,  we  report  a  summary  of  results  from  question  1  (which 
asked  respondents  to  identify  their  one  or  two  most  difficult  problems). 

Question  2 

A  summary  of  question  2  did  not  lend  itself  well  to  categorical  analysis.  In 
many  cases,  the  change  being  suggested  was  directly  related  to  the 
response  that  the  participant  provided  for  question  1 . 


To  analyze  and  quantify  the  narrative  information,  we  did  the  following: 

1 .  Categorized  each  response  by  tagging  it  with  a  key  word  or  short 
phrase. 

2.  Grouped  related  responses  as  guided  by  the  assigned  tag. 

3.  Reviewed  the  results  of  question  2  to  validate  the  category 
assignments  of  each  response. 

4.  Tallied  the  frequency  of  responses  within  each  category. 

5.  Charted  the  results  of  question  4. 

The  results  of  the  analysis  are  displayed  in  Figure  72  on  the  next  page. 
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Response  Categories 


Figure  72:  Frequency  of  top  two  problem  categories  identified  by  respondents 
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Example 

responses; 

Requirements 

Management 


Without  violating  the  anonymity  of  the  participants,  here  are  some  actual 
examples  of  the  narrative  responses  provided  in  the  survey: 

•  Inadequately  defined  requirements:  As  requirements  changed,  the 
new  requirements  were  poorly  defined  compounding  the  problems. 

•  Unclear  requirements. 

•  Definitive  requirements  identification. 

•  Requirements  creep. 

•  Changing  requirements/priorities. 

•  Defining  SW  requirements. 

•  Requirements  creep  due  to  changing  user  requirements. 

•  Requirements  creep  with  user  community  -  PM’s  must  be  able  to 
hold  the  line  to  their  customers  on  requirements,  complete  a  product, 
then  get  the  additional  requirements  in  a  follow-on  product. 


Example 

responses: 

Project 

Management 


Without  violating  the  anonymity  of  the  participants,  here  are  some  actual 
examples  of  the  narrative  responses  provided  in  the  survey: 

•  Software  sizing  and  cost  estimation. 

•  Maintaining  a  known  budget  from  which  to  plan  and  execute. 
Unanticipated  decrements,  regardless  of  reason,  continue  to 
challenge  the  Program  Office  and  developer  to  provide  a  product  on 
time,  within  cost  that  meets  all  agreed  to  specifications. 

•  Sometimes  very  difficult  to  synchronize  sub-component  software 
development  schedules  with  the  end  user  platform  software 
development  schedules.  Schedule  slips  have  ripple  effects  that 
require  intensive  management  to  overcome. 

•  Under-estimation  of  effort. 

•  Inaccurate  software  estimate  and  schedule — lack  of  historical  data 
from  developers. 

•  Program  schedule  requirements. 

•  Conflicting  schedules  among  systems  that  we  must  interface  with. 
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8  Observations  and  Conclusions 


Introduction 


Cannot  generalize 
the  results 


Biased 

responses 


The  survey  results  are  informative  and  provide  a  needed  starting  point  for 
further  investigations.  However,  there  is  a  risk  of  over-interpreting  the  results. 
For  example,  due  to  constraints  beyond  the  control  of  the  researchers,  it  was 
not  possible  to  characterize  the  target  audience  sufficiently  in  order  to  design 
a  sampling  plan  that  would  allow  determination  of  statistical  significance  for 
the  findings. 

In  addition,  there  are  other  threats  to  validity  that  should  be  considered,  so 
that  the  reader  can  couch  any  conclusions  about  the  results  within  a 
reasonabie  context. 


Typically,  survey  research  Is  conducted  when  we  would  like  to  know 
something  about  a  large  number  of  people  (the  population),  but  can  only  study 
a  small  number  of  them  (the  samp/e).  In  other  words,  the  intent  is  to  study  the 
sample  in  order  to  make  generalizations  about  the  population. 

Empirical  researchers  and  statisticians  often  speak  of  the  “vaiidity"  of  a  study. 
The  validity  of  a  study  refers  to  the  approximate  truth  of  propositions, 
inferences  and  conclusions. 

External  validity  refers  to  the  approximate  truth  of  conclusions  that  involve 
generalizations  (that  is,  generalizations  made  about  the  population  from  the 
information  generated  by  the  sample).  To  generalize  for  a  population,  surveys 
must  follow  strict  procedures  in  defining  whom  to  study  and  how  to  select 
them.  For  example,  a  random  sample  must  be  drawn  from  a  well-defined 
population.  Doing  so  enhances  the  probability  that  the  sample  is 
representative  of  the  population. 

In  this  study,  we  were  not  able  to  characterize  either  the  population  or  the 
sample  in  sufficient  detail  to  allow  us  to  make  generalizations  about  the 
opinions  of  the  population  of  Army  acquisition  managers.  Respondents  self- 
selected  themselves  to  participate  in  this  study  and  we  are  unable  to  establish 
if  the  sample  was  indeed  representative  of  the  population  of  interest. 
Therefore,  the  reader  should  be  aware  of  this  threat  to  external  validity  and 
that  making  generalizations  based  on  the  survey  results  is  risky  at  best. 


One  source  of  possible  inaccuracy  in  surveys  has  to  do  with  the  tendency  of 
individuals  to  offer  socially  desirable  answers  in  a  way  that  conforms  to 
dominant  belief  patterns. 

To  a  large  degree  in  this  survey,  program  managers  were  being  asked  to 
report  on  aspects  of  their  program  management  competencies.  If  the 
respondent  felt  that  their  answer  would  be  known  to  their  superiors,  then  they 
may  have  biased  their  response  to  gain  approval  in  the  eyes  of  people  whose 
approval  they  value. 

We  attempted  to  mitigate  this  risk  by  requesting  accurate  information  and 
assuring  the  participants  that  their  responses  would  not  be  attributed  to  any 
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individual  or  organization.  We  hope  that  this  was  sufficient  mitigation.  Yet,  in 
general,  the  researcher  was  struck  by  the  positive  way  in  which  a  majority  of 
respondents  characterized  their  use  of  effective  management  practices. 


Terminology 

issues 


False 

respondents 


Cause  and  effect 


For  any  survey,  the  wrong  choice  of  words  can  create  any  number  of 
problems— from  excessive  vagueness  to  too  much  precision,  from  being 
misunderstood  to  not  being  understood  at  all. 

To  mitigate  the  risk  of  misinterpretation  of  terminology  in  our  survey  items,  we 
conducted  reviews  with  acquisition  subject  matter  experts  and  members  of 
the  target  population.  The  reviewers  were  specifically  requested  to  identify 
any  terminology  that  could  be  misinterpreted  by  members  of  the  target 
population.  The  survey  instrument  was  revised  to  eliminate  these  types  of 
problems. 


For  this  self-administered  survey,  it  was  not  possible  to  verify  who  the  actual 
respondent  was.  The  intended  target  audience  was  Army  acquisition  program 
managers  who  are  responsible  for  software-intensive  systems.  Individuals 
self-selected  themselves  for  participation  in  this  survey.  It  appears  that  in 
some  cases,  the  program  manager  may  have  delegated  the  task  of 
completing  the  survey  to  others  in  their  organization.  Figure  3  on  page  15 
indicates  that  17%  of  those  responding  may  not  be  acquisition  managers. 

In  these  cases,  it  is  likely  that  the  task  was  delegated  by  the  program 
manager  to  those  they  perceived  were  more  qualified  to  answer  questions 
about  the  software  side  of  acquisition.  Since  the  purpose  was  really  to  obtain 
factual  insight  into  software  acquisition-related  issues,  having  these 
individuals  answer  for  their  managers  is  not  viewed  here  as  especially 
problematic. 


In  general,  it  would  be  erroneous  to  attribute  cause  and  effect  relationships  to 
results  obtained  in  this  survey.  Due  to  the  broadness  of  issues  covered  in  this 
survey  and  the  inherent  inability  of  surveys  to  delve  deeply  into  complex 
issues,  causal  relationships  could  not  be  explored. 

Further  investigations  would  be  required  to  establish  causal  relationships 
related  to  the  outcomes  of  this  survey. 
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8.1  Relevancy  of  Current  DoD  Acquisition-Based  Initiatives 

The  top  rated  DoD  We  asked  respondents  to  rate  the  relevancy  of  current  DoD  acquisition- 

initiatives  based  initiatives  (see  Figure  16  on  page  31).  Out  of  18  initiatives,  the  three 

highest  rated  were  (1)  interoperability,  (2)  performance  specifications,  and  (3) 
open  systems  design. 

In  response  to  an  open-ended  question,  individuals  were  asked  to  identify 
one  or  two  of  the  most  difficult  problems  that  their  project  has  faced  in 
conducting  successful  acquisition  of  software-intensive  systems. 

Interoperability  was  identified  in  7%  of  the  responses.  It  was  the  6"’  most- 
identified  problem  (see  Figure  72  on  page  101). 

In  December,  2002,  the  ASSIP  Action  Group  (AAG)  members  were 
administered  a  questionnaire  and  asked  the  very  same  question  about  the 
relevancy  of  current  DoD  acquisition-based  initiatives.  They  also  responded 
that  the  interoperability  initiative  was  the  most  relevant  to  how  they  perform 
their  work. 

Figure  73:  Comparison  of  survey  respondents  and  AAG  member  respondents 
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8.2  The  Acquirer  Environment  and  Communication 

Areas  There  were  a  number  of  areas  that  were  explored  within  the  acquirer 

investigated  environment  and  communication  area.  The  categories  included 

•  education  and  training 

•  software  acquisition  planning  and  project  management 

•  use  of  measurement  to  support  decision  making 

•  solicitation 

•  software-related  contractual  requirements  development  and 
management 

•  contract  performance  management 

•  transition  to  support 

•  transition  to  operations 


Comment  In  general,  the  survey  items  for  these  categories  were  worded  in  such  a  way 

that  a  respondent’s  agreement  (with  the  survey  item)  would  indicate  that  an 
effective  practice  was  in  use  within  the  program. 

Practices  that  are  The  following  table  presents  the  top  eight  survey  statements  that  were 

not  broadly  viewed  unfavorably  in  the  sense  that  the  stated  practice  is  not  in  use  within 

implemented  the  program. 


ID 

Survey  Item 

Agree 

1 

Use  of  Measurement  to  Support  Decision  Making: 

Your  project  team  records  statistical  and  quality  management 
data  in  the  organization’s  measurement  repository  and  uses 
that  information  for  decision  making. 

50 

2 

Software  Acquisition  Planning  and  Project  Management: 

There  is  a  change  request  process  for  submitting  suggestions 
for  improving  the  acquisition  process. 

56 

3 

Solicitation: 

Software  risks  are  independently  evaluated  as  part  of  the 
solicitation  and  are  communicated  to  the  solicitation  official. 

58 

4 

Education  and  Training: 

You  use  organizational  or  project  training  plans  to  plan  for 
individual  training. 

59 

5 

Software  Acquisition  Planning  and  Project  Management: 

A  novice  engineer  (participating  on  this  acquisition  project) 
would  know  how  to  surface  risks  according  to  the  risk 
identification  and  analysis  plan. 

59 

Table  continues  on  next  page 
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Practices  that  are  This  table  continues  from  previous  page. 

not  broadly 
implemented, 

continued  _ _ 

ID  Survey  Item 

6  Software-Related  Contractual  Requirements  66 

Development  and  Management: 

A  member  of  the  acquisition  project  staff  or  a  novice  engineer 
could  identify  and  verify  the  source  of  software-related 
contractual  requirements. 

7  Contract  Performance  Management:  67 

Your  project  team  identifies,  documents,  and  tracks  software 
risks  associated  with  the  contractor’s  efforts,  independent  of 
the  contractor’s  risk  management  process. 

8  Transition  to  Support:  66 

The  strategy  for  transition  into  maintenance  is  documented, 
communicated,  and  agreed  to  by  all  parties  early  in  the 
acquisition. 


Of  the  items  listed  in  the  table  above,  the  following  are  of  interest: 

•  Three  of  the  top  eight  items  above  (IDs  3, 5,  and  7)  are  related  to  the 
use  of  risk  management  procedures.^ 

•  Two  of  the  items  (IDs  5  and  6)  include  the  term  “novice  engineer”  as 
part  of  the  conditions  associated  with  implementation  of  the  practice. 
Only  three  items  in  the  entire  survey  used  this  term. 

The  use  of  this  term  may  have  affected  how  individuals  responded  to 
this  question.  It  might  imply  that  the  practice  is  not  well-enough 
documented  so  that  a  new  employee  could  be  apprised  of  how  this 
practice  is  implemented.  If  the  term  was  not  included,  one  might 
speculate  that  the  percentage  in  agreement  (with  the  statement)  may 
have  been  higher. 

•  It  is  not  surprising  that  many  respondents  did  not  agree  with  item  ID 

1  in  the  table.  This  item  refers  to  the  use  of  quantitative  measures  for 
decision  support  and  the  use  of  an  organizational  measurement 
repository.  What  is  surprising  is  that  50%  of  the  respondents  stated 
that  they  do  use  such  mechanisms.  Use  of  such  processes  and 
mechanisms  are  typically  associated  with  a  very  disciplined 
organization  with  mature  processes.* 


^  It  should  also  be  noted  that  of  42  items  addressed  in  this  section  (that  is,  “The  Acquirer  Environment  and 
Communication”),  8  of  those  items  addressed  aspects  of  risk  management. 

For  example,  these  types  of  effective  practices  are  associated  with  Levels  4  and  5  of  the  Software  Acquisition 
Capability  Maturity  Model®  (SA-CMM^  {CMU/SEI-2002-TR-010). 
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Comparing  topical 
categories 


In  the  table  below,  the  survey  item  category  is  listed  on  the  ieft.  To  the  right 
are  three  columns  that  list  the 

•  lowest  percent  (of  respondents)  agreeing  with  any  single  survey  item 
from  that  category: 

•  average  percent  of  respondents  in  agreement  with  all  survey  items 
from  a  category;  and  the 

•  highest  percent  (of  respondents)  that  agree  with  any  single  survey 
item  in  that  category. 


Percent  that  agree  w/  survey  Items 


Low 

Average 

High 

Education  and  training 

59 

75 

88 

Software  acquisition  planning  and  project 
management 

56 

76 

87 

Use  of  measurement  to  support  decision 
making 

50 

73 

83 

Solicitation 

58 

67 

80 

Software-related  contractual  requirements 
development  and  management 

66 

75 

79 

Contract  performance  management 

67 

77 

86 

Transition  to  support 

66 

69 

70 

Transition  to  operations 

70 

80 

85 

These  values  can  be  used  to  conduct  a  crude  comparison  of  how 
respondents  reacted  to  questions  from  one  category  to  the  next.  However, 
this  comparison  must  be  done  with  caution,  since  it  is  impossible  to  draw 
specific  conclusions  without  inspecting  the  wording  of  specific  survey  items  in 
each  of  the  categories. 

If  one  looks  across  the  categories  of  survey  items,  it  is  noted  that  Solicitation 
was  the  category  that  appeared  to  have  the  most  unfavorable  responses  on 
average  (in  terms  of  the  indicated  practices  being  implemented).  The 
questions  within  ‘Transition  to  supporf’  also  indicate  a  lower  agreement  level 
when  compared  to  the  other  categories  of  survey  items. 

While  the  results  of  this  analysis  are  by  no  means  conclusive,  they  do  point  to 
areas  that  should  be  investigated  in  more  depth  to  expose  potential  high- 
impact  improvement  opportunities. 
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Use  of 

measurement 


Respondents  were  asked  to  identify  whether  they  were  using  measurement 
to  track  program  status  and  progress.  The  table  below  is  reproduced  from 
chapter  7,  Overview  of  Survey  Results. 


Measure 

Percent 

using 

Requirements  stability 

51 

Manpower 

63 

Development  progress 

82 

Cost 

86 

Schedule 

91 

It  is  interesting  to  note  that,  in  particular,  a  large  percentage  of  managers  are 
not  tracking  requirements  stability. 

This  result  is  noted  in  relation  to  a  different  question  posed  to  participants. 
When  respondents  were  asked  to  identify  the  two  most  problematic  areas 
that  they  must  contend  with,  requirements  management  was  identified  as  the 
top  item.^  (See  Figure  72  on  page  101 .) 

Perhaps  the  lack  of  measurement  and  tracking  of  requirements  is  part  of  the 
requirements  stability  problem.  If  measures  are  used  to  obtain  a  better 
understanding  of  the  issues  related  to  requirements  stability,  then  this  could 
lead  to  identification  of  appropriate  improvement  strategies. 

With  respect  to  the  other  measures  reported  above,  it  is  surprising  that  such 
high  proportions  of  the  respondents  are  nof  tracking  such  fundamental 
project  characteristics  as  schedule  and  cost. 


^  In  response  to  an  open-ended  question  regarding  the  top  two  problems  faced  by  program  managers,  28%  of  the 
responses  identified  requirements  management.  Project  management  (10%)  and  contractor’s  processes  (10%) 
were  the  other  top  problems  identified. 
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Requirements  In  this  section  of  the  survey,  seven  close-ended  questions  addressed  aspects 

management  of  requirements  management*.  In  general,  the  response  pattern  was  rather 

normal  when  compared  to  responses  from  other  categories.  Exceptions 
included  responses  to  the  following  questions: 


Survey  Item 

%  Agree 

The  software-related  contractual  requirements  baseline  is 
established  prior  to  release  of  the  solicitation  package. 

51 

A  member  of  the  acquisition  project  staff  or  a  novice 
engineer  could  identify  and  verify  the  source  of  software- 
related  contractual  requirements. 

63 

When  responding  to  the  open-ended  question  regarding  the  two  top 
problems  that  the  respondent  encounters  in  acquisition,  requirements 
management  was  identified  most  often  as  the  top  problem  encountered. 


Project  When  responding  to  the  close-ended  survey  items  that  addressed  program 

management  planning  and  project  management,  respondents  seemed  to  indicate  that 

processes  these  processes  were  generally  implemented  in  a  sufficient  way  (see  page 

42). 

Yet,  when  asked  to  identify  the  top  two  problems  that  they  faced,  a  large 
number  of  responses  were  associated  with  project  management.  Project 
management  was  identified  in  1 0%  of  the  open-ended  responses  (see  page 
101). 

This  may  or  may  not  present  an  inconsistency.  One  possibility  is  that  the 
close-ended  questions  did  not  address  the  project  management  concerns  of 
managers.  Another  possibility  is  that  although  the  close-ended  response 
profile  didn’t  show  major  irregularities,  perhaps  the  managers  who  responded 
that  they  don’t  apply  effective  practices  are  reflecting  (in  their  open-ended 
responses)  that  these  are  significant  problems  for  their  program. 


Two  requirements  management  related  close-ended  questions  were  in  the  “Solicitation”  category  and  five  questions 
were  in  the  Software  Related  Contractual  Requirements  Development  and  Management  category. 
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8.3  The  Developer’s  Environment 

Rating  the  Respondents  were  asked  to  rate  the  system  developer’s  abilities  in  a  number 

developer’s  of  key  process  areas.  The  scaled  results  are  presented  graphically  in  Figure 

processes  60  and  reproduced  in  table  form  below.  The  scale  assignments  were  defined 

as  follows: 


Excellent 

=  5  - 

Below  Average 

=  1 

Above  Average 

=  3.5 

Extremely  Poor 

=  0 

Average 

=  2.5 

Not  relevant  or  don’t  know 

=  0 

System  Developer’s  Processes 

Rating 

Technical  solution 

2.8 

Project  monitoring  and  control 

2.7 

Software  architecture  development  and  assessment 

2.7 

Product  integration 

2.7 

Configuration  management 

2.7 

Integrated  teaming 

2.7 

Sharing  all  relevant  information  that  you  feel  you  should 
know  to  manage  the  acquisition  effectively 

2.7 

Project  planning 

2.6 

Supplier  or  subcontract  management 

2.6 

Defined  processes  that  support  product  development  and 
stable  operations 

2.6 

Requirements  development 

2.5 

Requirements  management 

2.5 

Verification 

2.5 

Measurement  analysis  and  reporting 

2.4 

Validation 

2.4 

Causal  analysis  and  resolution  for  defect  prevention 

2.2 

Risk  management 

2.0 

When  rating  the  system  developer’s  processes,  acquisition  mangers  (who 
responded)  seem  to  perceive  that  performance  is  average  on  a  scale  from 
excellent  (5)  to  extremely  poor  (0).  Whether  these  perceptions  are  based  on 
first-hand  knowledge  or  through  other  means  is  unknown  to  the  researcher. 

it  is  interesting  to  note  that  performance  was  rated  lowest  for  risk 
management.  This  is  an  area  in  which  a  proportion  of  respondents  have 
implied  deficiencies  on  their  own  part  (see  page  106). 
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8.4  Impact  of  External  Factors  on  Acquisition 

Key  results  Six  close-ended  survey  items  addressed  the  impact  of  external  factors  on 

the  success  of  the  respondent’s  acquisition  process.  The  results  are 
presented  in  Figures  61-66  beginning  on  page  88. 

In  particular,  the  responses  from  two  questions  in  this  section  of  the 
survey  draw  special  attention.  They  appear  below,  and  the  right  column 
shows  the  percent  of  respondents  who  agreed  with  the  statement. 


Survey  Item 

%  Agree 

Reallocation  of  program  funding  is  a  significant  source 
of  frustration  in  acquisition  programs. 

90 

Criticai  personnel  are  lost  due  to  military  rotations  or 
inability  to  compete  with  industry  salaries. 

55 

Note  that  90%  of  the  respondents  feel  that  reallocation  of  program  funding 
is  a  significant  source  of  frustration.  Ninety  percent  agreement  (for  a 
survey  item)  is  the  highest  level  of  agreement  among  the  respondents 
being  reported  as  part  of  these  results. 

Fifty-five  percent  or  nearly  half  of  the  respondents  believe  that  personnel 
turnover  may  be  a  source  of  problems  in  their  acquisition  environment. 


112 


CMU/SEI-2004-TR-003 


8.4.1  Where  and  What  Are  the  Major  Problems  and  Risks 


Where? 


Respondents  were  asked  to  identify  where  they  perceived  the  major  problems  and 
risks  in  their  acquisition  programs  to  be.  The  areas  considered  are 


1 .  acquisition  program  policies  and  processes 

2.  the  contracting  process  between  acquirers  and  developers 

3.  the  contractor’s  (or  supplier’s)  development  process 

4.  factors  outside  the  control  of  the  acquirers  and  developers  (congressional 
mandates,  priorities  set  by  engagements  of  our  armed  forces,  etc.) 

m 


Factors  external  to  Acquisition  System  that  impact  it 


r  Acquirer 
i  Environment 


Communication 


L--.  :  . 

The  Acquisition  System 


Developer  : 
i  Environment 


For  each  of  the  four  areas,  respondents  were  asked  to  indicate  the  degree  to 
which  problems  and  risks  can  be  attributed  to  an  area  in  the  following  way: 

•  To  a  great  extent  (5) 

•  Somewhat  (3) 

•  Very  little  (1 ) 

•  Not  at  all  (0) 

The  results  are  presented  in  Figures  67-71.  A  summary  is  presented  in 
tabular  form  below. 


Survey  Item 

Score 

Acquirer  processes 

2.9 

Communication 

2.8 

Developer  processes 

2.8 

Factors  external  to  acquisition  system  that  impact  it 

3.0 

The  results  are  quite  similar  for  each  area  (2.8-3.0).  In  addition,  the  response 
profiles  for  each  area  (Figures  68-71)  are  remarkably  similar.  As  a  whole, 
respondents  believe  that  major  problems  and  risks  are  only  somewhat 
attributable  to  a  specific  area. 
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What? 


Requirements 

Management 


Respondents  were  asked  to  identify  what  they  perceived  as  the  major 
problems  and  risks  in  their  acquisition  programs.  The  following  open-ended 
question  was  included  in  the  survey: 

Overall,  what  are  the  one  or  two  most  difficult  problems  that  your 
project  has  faced  in  conducting  successful  acquisition  of  software¬ 
intensive  systems?  (Please  describe) 

The  open-ended  responses  were  classified  by  recurring  themes  using  an 
adaptation  of  the  affinity  grouping  technique. 

The  results  are  reported  in  Figure  72  and  a  summary  of  the  results  is 
reproduced  below. 


Total  number  of  responses 

225 

Requirements  management 

28% 

Project  management 

10% 

Contractor  processes 

10% 

Unstable  funding 

10% 

Required  skills 

7% 

Interoperability 

5% 

Requirements  management  was  most  often  identified  as  a  major  problem 
and  risk  area.  This  result  is  consistent  with  responses  from  the  following 
close-ended  survey  items: 


Survey  item 

%  Agree 

The  software-related  contractual  requirements  baseline  is 
established  prior  to  release  of  the  solicitation  package. 

51 

A  member  of  the  acquisition  project  staff  or  a  novice 
engineer  could  identify  and  verify  the  source  of  software- 
related  contractual  requirements. 

63 

Such  low  agreement  percentages  for  these  statements  seem  to  validate  the 
notion  that  requirements  management  represents  a  potential  high-impact 
area  for  future  improvement  activities. 
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Project 

Management 


Contractor 

processes 


Unstable 

funding 


Required 

skills 


Interoperability 


Ten  percent  of  the  responses  suggested  that  project  management  was  the 
source  of  most  problems  and  risks. 

In  response  to  close-ended  survey  items  addressing  project  management, 
the  data  indicates  that  a  rather  large  majority  (approximately  80%)  of 
respondents  feel  that  effective  practices  are  being  used  (see  page  39). 

One  might  conjecture  that  for  those  who  do  have  project  management 
issues,  they  represent  a  significant  obstacle  to  accomplishing  successful 
acquisition  projects. 


Ten  percent  of  the  responses  identified  contractor  processes  as  a  major 
problem  or  risk  area.  When  asked  to  rate  the  contractor’s  technical 
processes,  the  respondents  reported  that  the  overall  performance  was  close 
to  average  (on  a  scale  from  “excellent”  to  “very  poor”).  (See  Figure  60  on 
page  86.) 

As  with  the  project  management  area,  this  may  indicate  that  for  the 
proportion  of  respondents  who  did  report  a  problem  or  issue  with  contractor 
processes,  they  believe  that  it  impacts  them  in  a  significant  way. 


Ten  percent  of  the  responses  identified  unstable  funding  as  a  major  problem 
or  risk  area. 

It  is  not  a  surprise  that  this  issue  showed  up  in  the  top  5  issues  reported  by 
the  participants  in  response  to  this  open-ended  question.  Ninety  percent  of 
respondents  indicated  agreement  with  this  statement: 

Reallocation  of  program  funding  is  a  significant  source  of 
frustration  in  acquisition  programs. 

This  appears  to  be  an  issue  for  many  Army  acquisition  managers. 


Required  skills  was  identified  as  a  problem/risk  issue  in  7%  of  the  responses. 

When  examining  the  actual  statements  that  were  reported,  the  issue  is  that 
there  is  an  insufficient  number  of  people  with  expertise  in  software 
engineering. 

Also,  55%  of  respondents  indicated  that  critical  personnel  are  lost  due  to 
military  rotations  or  the  inability  of  the  acquisition  organization  to  compete 
with  industry  salaries. 


Five  percent  of  the  responses  identified  interoperability  as  a  major  problem  or 
risk  area.  This  is  not  surprising  given  that  it  is  a  well-known  challenge  area 
across  the  services. 

Among  current  DoD  acquisition-based  initiatives,  interoperability  was 
identified  as  the  most  relevant  initiative  to  how  respondents  perform  their 
work.  See  Figure  16  on  page  31 . 
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9  Major  Themes 


Introduction  In  this  chapter,  the  major  improvement  priority  themes  are  summarized  from 

chapter  8  on  page  1 03. 


Primary 

Theme: 

Requirements 

Management 


Based  on  the  results  of  this  survey,^  requirements  management  is  a  primary 
theme.  Evidence  of  this  includes  the  following: 

•  This  area  was  identified  most  often  as  the  cause  of  the  majority  of 
problems  and  risks  to  project  managers  (see  Figure  72  on  page  101). 

•  Responses  to  two  close-ended  questions  indicated  issues  with 
this  area  (see  the  “Requirements  management”  heading  in 
section  8.2). 

•  Approximately  50%  of  respondents  report  that  they  do  nof  track 
requirements  stability  (see  the  “Use  of  measurement”  heading  in 
section  8.2). 

A  large  proportion  of  the  respondents  clearly  indicate  that  this  area  is 
problematic  for  them. 


Secondary  In  addition  to  requirements  management,  there  was  evidence  that  additional 

themes  areas  should  be  more  thoroughly  investigated  as  potential  areas  for 

improvement  interventions.  These  areas  include 

•  unstable  funding 

•  risk  management 

•  interoperability 

•  solicitation 

•  transition  to  support 

•  project  management 

•  required  skills 

The  basis  for  identifying  these  areas  as  secondary  themes  is  presented  on 
the  next  page. 


^  It  has  already  been  stated  in  this  report  that  the  results  of  this  survey  cannot  be  generalized  to  the  overall 
population  of  Army  acquisition  managers  who  are  responsible  for  software  intensive  systems.  See  chapter  10  on 
page  1 1 9  for  a  discussion  of  validity  issues  and  interpretation  guidelines  for  the  information  presented  in  this  report. 
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Evidence; 

Secondary 

Themes 


The  basis  for  selecting  the  secondary  themes  is  listed  below. 

Unstable  Funding 

•  Ninety  percent  of  respondents  agree  that  this  area  is  a  source  of 
frustration  (see  section  8.4  on  page  112). 

•  One  of  the  top  five  problems/risks  identified  by  respondents  (see 
section  8.4.1  on  page  113). 

Risk  Management 

•  Three  of  the  top  eight  survey  items  that  were  identified  as  practices 
not  implemented  were  associated  with  risk  management  (see 
“Practices  that  are  not  broadly  implemented”  in  section  8.2  which 
begins  on  page  106). 

•  Risk  management  was  perceived  as  the  weakest  technical 
process  of  developers  (see  section  8.3  on  page  111). 

Interoperability 

•  Identified  as  the  most  relevant  among  all  other  current  DoD 
acquisition-based  initiatives  (see  Figure  16  on  page  31). 

•  Identified  as  one  of  the  top  five  problems/risks  identified  by 
respondents  (see  “What”  in  section  8.4.1  which  begins  on  page 
113). 

Solicitation 

•  When  comparing  responses  to  categories  of  survey  items, 
solicitation  was  the  category  that  appeared  to  have  the  most 
unfavorable  responses  on  average  (see  “Comparing  topical 
categories”  in  section  8.2  which  begins  on  page  106). 

Transition  to  Support 

•  When  comparing  responses  between  categories  of  questions, 
transition  to  support  was  a  category  that  appeared  to  also  have 
the  most  unfavorable  responses  on  average  (see  “Comparing 
topical  categories”  in  section  8.2  which  begins  on  page  106). 

Project  Management 

•  This  area  was  identified  in  10%  of  responses  as  a  major 
problem/risk  area  (see  “What”  in  section  8.4.1  which  begins  on 
page  1 1 3).  But  responses  to  close-ended  questions  in  this  area 
didn’t  provide  strong  validation  that  project  management 
represents  a  pervasive  problem  in  the  Army  Acquisition  System. 

Required  skills 

•  Identified  as  one  of  the  top  five  problems/risks  identified  by 
respondents  (see  “Whaf  in  section  8.4.1  which  begins  on  page  113). 

•  Fifty-five  percent  of  respondents  state  that  turnover  of  critical 
personnel  is  an  issue  (see  section  8.4  on  page  112). 
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10  Summary 


Limitations  of 
surveys 


Themes 


Benefits 


Next  steps 


^  See  page  100-101. 


Survey  data  is  always  somewhat  superficial.  It  is  not  possible  to  go  into  great 
detail  or  address  complicated  questions.  Surveys  are  not  capable  of  digging 
deeply  into  people’s  psyches  looking  for  fundamental  explanations  of  their 
unique  understandings  or  behaviors.  (This  was  especially  the  case  for  this 
survey,  which  investigated  a  broad  set  of  issues  spanning  the  Army 
acquisition  system.) 

For  these  reasons  and  others  that  have  been  addressed  in  this  paper, ^  the 
reader  should  be  careful  not  to  over-interpret  the  results  reported  in  this 
paper. 

This  survey  should  be  considered  a  single  data  point  in  an  overall 
information-gathering  approach  that  includes  multiple  on-site  interviews  at 
Army  acquisition  program  sites.  The  ASSIP  strategy  is  to  triangulate  from 
these  multiple  data-gathering  approaches  to  paint  a  more  reliable  picture  of 
the  key  characteristics  of  the  system.  Up  to  10  in-depth  on-site  interviews  are 
planned  for  fiscal  year  2004. 


Major  themes  have  been  identified  based  on  synthesizing  the  information 
from  this  survey.  These  themes  are  defined  in  rather  broad  terms  chapter  9. 
These  themes  should  be  considered  areas  requiring  more  detailed 
investigation  as  potential  areas  for  improvement  interventions. 


The  information  obtained  in  this  survey  provides  an  important  foundation  for 
future  data-gathering  events.  We  d/d  obtain  very  useful  information. 

The  results  will  significantly  enhance  our  ability  to  home  in  on  key  areas  of 
interest  that  should  be  the  focus  of  future,  detailed,  information-gathering 
events.  As  the  layers  of  the  acquisition  system  are  peeled  back,  we  will  get 
closer  to  the  root  causes  of  the  problem  areas  that  have  been  identified  here 
at  a  thematic  level. 


The  survey  results  will  be  used  to  inform  the  data-gathering  objectives  of  the 
depth  interviews  to  be  conducted  at  Army  acquisition  sites  during  fiscal  year  2004. 

The  information  obtained  from  this  survey  will  be  combined  with  the 
outcomes  of  the  depth  interviews,  and  the  triangulated  results  will  be 
synthesized. 

The  ASSIP  plan  envisions  future  surveys  as  a  way  to  continually  expand  our 
understanding  of  the  state  of  the  Army  Acquisition  System.  Information  from 
these  surveys  (as  well  as  the  depth  interviews)  will  lead  to  a  progressively 
more  detailed  and  well-understood  depiction  of  the  Army  Acquisition  System 
so  that  high-impact  improvement  interventions  can  be  planned  effectively. 
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Feedback  or 
questions 


Please  direct  any  feedback  or  questions  about  this  document  to 
Mark  Kasunic 

Software  Engineering  Management  and  Analysis 
Software  Engineering  Institute 
Carnegie  Mellon  University 

Email:  mkasunic@sei.cmu.edu 
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Appendix  -  Survey  Questionnaire 


In  this  section 


The  survey  questionnaire  instrument  is  presented  in  this  appendix,  starting 
on  the  next  page. 

This  is  the  paper  form  of  the  questionnaire  that  was  used  by  several 
respondents  who  were  unable  to  access  the  survey  instrument  over  the 
Internet  via  the  World  Wide  Web. 
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Appendix  -  Survey  Questionnaire 


Army  Strategic  Software  Improvement  Program  (ASSIP) 

“  “  Survey  of  Program  Managers 


Call  to  action  Any  effective  drive  begins  with  effective  intelligence  about  the  real  context  of  the 

overall  engagement.  For  this  reason,  it  is  our  intention  that  this  improvement 
initiative  begin  with  a  capture  of  the  reality  of  your  real  work  environment. 

This  survey  is  the  first  step  in  a  process  that  will  result  in  changes  to  your 
environment.  Our  effectiveness  in  acquiring  an  accurate  picture  of  the  real  issues 
in  Army  acquisition  is  dependent  on  your  careful  participation  in  this  survey. 

Therefore,  please  complete  this  survey  according  to  the  instructions  at  your 
earliest  convenience. 

Complete  by  Please  complete  this  survey  by  7  April  2003.  We  have  an  ambitious  schedule  for 

our  improvement  effort  and  your  cooperation  in  this  regard  is  vital  to  our  success. 


Confidentiality  Your  answers  will  be  held  in  the  strictest  of  confidence.  Information  that  can 

identify  you  and  your  organization  will  be  used  for  administrative  purposes  of  this 
effort  only.  Your  answers  will  be  used  in  summary  statistical  form.  Specific 
answers  will  never  be  identified  by  organization  or  individual. 


Purpose  This  survey  includes  questions  about  your  experiences  in  acquiring  software¬ 

intensive  systems  for  the  Army.  The  results  of  this  survey  will  be  used  to  formulate 
plans  for  improving  the  acquisition  system.  Results  from  the  survey  will  be  used  as 
a  means  of  gaining  insight  into  the  processes  of  acquisition.  The  insights  will  then 
be  used  to  identify  high-leverage  points  of  entry  for  improvement  activities  to  follow 
from  the  analysis  of  survey  results. 


Background  The  Army  Strategic  Software  Improvement  Program  (ASSIP)  was  initiated  by  the 

Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics  and  Technology 
(ASA(ALT)),  Mr.  C.  Bolton,  as  a  mechanism  for  improving  the  acquisition  of 
Software  Intensive  Systems  (SIS). 

One  of  the  major  goals  of  the  ASSIP  for  FY  03  is  to  develop  a  Strategic  Software 
Improvement  Master  Plan  (SSIMP).  In  order  to  execute  the  ASSIP  and  develop 
the  SSIMP,the  ASSIP  establishes  a  Senior  Steering  Group  (SSG),  made  up  of 
PEOs  and  separate  reporting  PMs  and  a  supporting  ASSIP  Action  Group  (AAG), 
made  up  of  representatives  of  the  SSG  Members.  The  Software  Engineering 
Institutute  (SEI)  has  been  selected  to  assist  in  the  evolution  and  execution  of  the 
ASSIP  and  to  participate  in  support  of  both  established  bodies. 
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ASSIP 

Survey  of  Program  Managers 


About  you  and  your  acquisition  project 


1.1.  Please  indicate  the  role  that  best  describes  your  position  or  check  “other”  and  list  the  position. 

□  Program^  Executive  Officer  (PEO) 

□  Program  Manager  of  an  acquisition  program 

□  Project  or  Product  Manager 

□  Deputy  Program  Manager  for  an  acquisition  program 

□  Other  (please  describe) 


1 .2.  Please  indicate  the  selection  that  best  describes  your  classification. 

Military  Civilian 


□ 

Major  General 

□ 

SES 

□ 

Brigadier  General 

□ 

GS14-15 

□ 

Colonel 

□ 

GS13 

□ 

LTC 

□ 

GS9-12 

□ 

Other  (please  describe) 

1.3.  Years  of  experience  working  as  an  Army  Acquisition  Manager: 

□  0-2  years  □  6-10  years 

□  3-5  years  D  Over  10  years 


1 .4.  Years  of  experience  in  current  position: 

□  0-2  years  □  6-10  years 

□  3-5  years  □  Over  10  years 


1 .5.  How  would  you  describe  your  level  of  personal  expertise  in  each  of  the  areas  below? 


Extensive 

Substantial 

Moderate 

Little  if  any 

Acquisition  program/project  management 

□ 

□ 

□ 

□ 

Software  acquisition  management 

□ 

□ 

□ 

□ 

Software  engineering  technical  practices  (e.g., 
requirements  management,  design, 
configuration  control,  coding,  testing,  etc.) 

□ 

□ 

□ 

□ 

Systems  Engineering 

□ 

□ 

□ 

□ 

^  In  this  survey,  when  we  use  the  term  program  manager,  we  mean  program,  project  or  product  manager  or  project 
director. 
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Appendix  -  Survey  Questionnaire 


1 .6.  Indicate  the  number  of  programs  in  each  Acquisition  Category  (ACAT)  that  you  are  currently 
responsible  for. 


Number  of  Programs 

0 

1 

2 

3-5 

>5 

ACAT  I 

□ 

□ 

□ 

ACAT  II 

□ 

□ 

□ 

□ 

□ 

ACAT  III 

□ 

□ 

□ 

□ 

n 

Other 

□ 

□ 

□ 

□ 

□ 

1 .7.  Indicate  the  number  of  each  type  of  software-intensive  system  that  you  are  currently  responsible  for 
by  checking  the  appropriate  box  in  each  row  of  the  table. 


Number  of  Systems 

0 

1 

2 

3-5 

>5 

Automated  Information  Systems 

For  example,  management  information  systems  supporting 
business  operations  such  as  payroll,  inventory,  or  logistics. 

□ 

□ 

n 

□ 

□ 

Weapons  Systems 

For  example,  systems  with  real-time  process  control  or 
guidance  systems  for  avionics  or  radar;  embedded  software 
running  in  electronic  devices,  vehicles,  missiles  or  aircraft. 

□ 

□ 

□ 

□ 

□ 

C^IEW  or  C4ISR 

For  example,  decision  support  systems,  intelligence  systems, 
mission  planning,  communications  systems,  or  maneuver 
control. 

□ 

□ 

□ 

□ 

□ 

Other 

For  example,  modeling  and  simulation,  compilers,  configuration 
management  tools,  cost  estimation  tools,  personal  computer 

I  applications,  pattern  recognition,  expert  systems. 

□ 

□ 

□ 

□ 

□ 

124 


CMU/SEI-2004-TR-003 


About  Your  Acquisition  Project 


DoD  Acquisition-Based  initiatives 

c 

5 

10-  The  DoD  has  implemented  the  major  acquisition-based 

4-» 

c 

CO 

o 

2 

2 

O 

initiatives  identified  below.  Please  indicate  your  perception 

13 

c 

(0 

$ 

of  the  relevance  of  these  initiatives  to  how  you  perform 

0) 

c 

fB 

£1 

% 

> 

o 

o 

c 

your  work. 

> 

a> 

a> 

% 

E 

o 

a> 

"S 

•jc 

c 

o 

X 

cr 

w 

Q 

10.1 

Performance-based  Payments 

□ 

□ 

□ 

□ 

□ 

10.2 

Price-based  Acquisition 

□ 

□ 

□ 

□ 

□ 

10.3 

Performance-based  Services  Acquisition 

□ 

□ 

□ 

□ 

□ 

10.4 

Interoperability 

□ 

□ 

□ 

□ 

□ 

10.5 

Cost  As  An  Independent  Variable 

□ 

□ 

□ 

□ 

□ 

10.6 

Electronic  Business 

□ 

□ 

□ 

□ 

□ 

10.7 

Knowledge  Management  In  The  Acquisition  Workforce 

1  D 

□ 

1 

□ 

□ 

□ 

10.8 

Past  Performance 

□ 

! 

□ 

□ 

□ 

□ 

10.9 

Performance  Specifications 

□ 

□ 

□ 

□ 

□ 

10.10 

FAR  Part  12  /  Commercial  Item  Procurements 

□ 

□ 

□ 

□ 

□ 

10.11 

Total  Ownership  Cost  Management 

□ 

□ 

□ 

□ 

□ 

10.12 

ISO  9001 : 2000  Quality  Management  Systems 

□ 

□ 

□ 

□ 

□ 

10.13 

Open  Systems  Design 

□ 

□ 

□ 

□ 

□ 

10.14 

Integrated  Digital  Environment 

□ 

□ 

□ 

□ 

□ 

10.15 

Logistics  Transformation 

□ 

□ 

□ 

□ 

□ 

10.16 

Contractor  Incentives  Guide 

□ 

□ 

□ 

□ 

□ 

10.17 

Reducing  Government  Property  in  Possession  of  Contractors 

□ 

□ 

□ 

□ 

□ 

10.18 

Intellectual  Property  Guidelines 

□ 

□ 

□ 

□ 

□ 
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Appendix  -  Survey  Questionnaire 


Education  and  Training 

1 1 .  The  following  items  address  whether  your  organization  and  project 
team  develop  the  skills  and  knowledge  of  individuals  so  they  can 
perform  their  software  acquisition  roles  effectively  and  efficiently. 


11.1  Training  that  is  required  for  the  project  teams  to  achieve  their 
software  acquisition  objectives  is  identified  and  provided. 

1 1 .2  You  know  who  to  contact  when  you  have  training  needs  from 
the  organization. 

1 1 .3  There  are  mechanisms  in  place  that  allow  you  to  provide 
feedback  with  regard  to  the  effectiveness  of  training. 

1 1 .4  You  use  organizational  or  project  training  plans  to  plan  for 
individual  training. 

1 1 .5  In  general,  you  feel  there  are  ample  training  opportunities 
available  to  ensure  that  project  staff  have  the  right  skills  to 
perform  their  jobs. 
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Strongly  Disagree 


AppefKiiA  Survey 


Software  Acquisition  Pianning  &  Project  Management 

CD 

12.  The  following  items  address  whether  software  acquisition  planning 

CD 

CD 
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k. 
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is  conducted  in  a  way  that  is  useful  and  efficient  for  the  program, 
indicate  the  most  appropriate  response  to  each  of  the  following 
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12.1 

Software  experts  participate  in  system  acquisition  planning. 

□ 

□ 

□ 

□ 

□ 

12.2 

Acquisition  plans  are  revised  when  major  changes  occur. 

□ 

□ 

□ 

□ 

□ 

12.3 

Project-wide  participation  in  the  identification  and  mitigation  of 
risks  is  encouraged  and  valued  by  management. 

□ 

□ 

□ 

□ 

□ 

12.4 

Your  acquisition  project  assesses  the  likelihood  and 
consequence  of  each  risk  and  monitors  the  status  of  each  risk 

□ 

□ 

□ 

□ 

□ 

item. 

12.5 

A  novice  engineer  (participating  on  this  acquisition  project) 
would  know  how  to  surface  risks  according  to  the  risk 
identification  and  analysis  plan. 

□ 

□ 

□ 

□ 

□ 

12.6 

Weekly  or  biweekly  status  checks  or  other  periodic  reviews  are 
held  to  manage  and  control  risks,  issues  and  problems 
discovered  during  the  software  acquisition. 

□ 

□ 

□ 

□ 

□ 

12.7 

If  a  novice  engineer  discovers  a  problem  or  risk  in  the  system 
design,  1  am  confident  that  they  would  know  what  to  do  to 

□ 

□ 

□ 

□ 

□ 

surface  that  issue. 

12.8 

There  is  a  well-understood  and  effective  process  for  resolving 
issues  among  ail  project  functions. 

□ 

□ 

□ 

□ 

□ 

12.9 

There  is  a  change  request  process  for  submitting  suggestions 
for  improving  the  acquisition  process. 

□ 

□ 

□ 

□ 

□ 
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Appendix  -  Survey  Questionnaire 


Use  of  Measurement  to  Support  Decision-Making 

13.  The  following  items  address  whether  your  project  uses 

measurement  to  quantitatively  control  the  project’s  performance. 

Indicate  the  most  appropriate  response  to  each  of  the  following 
statements. 


13.1  Planning  estimates  are  based  on  historical  measurement  data 
from  previous  acquisition  projects. 

1 3.2  Measurement-based  objectives  for  the  acquired  products  and 
services  are  defined. 

1 3.3  The  acquisition  project  uses  metrics  as  an  input  to  program 
decision-making. 

1 3.4  The  performance,  cost,  and  schedule  objectives  of  the  software 
acquisition  project  are  measured  and  controlled  throughout  the 
software  acquisition. 

13.5  Your  project  team  uses  measures  and  analytic  techniques  for 
statistically  managing  selected  processes  and  sub-processes. 

1 3.6  Your  project  team  records  statistical  and  quality  management 
data  in  the  organization’s  measurement  repository  and  uses 
that  information  for  decision-making. 


14.  The  following  metrics  are  reported  to  the  PMO  on  (at  least)  a  monthly  basis. 

□  Cost 

□  Schedule 

□  Manpower 

□  Development  progress 

□  Requirements  stability 

□  Don’t  know 
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Strongly  Disagree 


Mpperi. 


Solicitation 

15.  The  following  items  address  whether  solicitation  packages  prepared 
by  the  program  specify  technical  needs  adequately,  and  whether 
the  proposal  review  process  succeeds  at  identifying  capable 
contractors. 

Indicate  the  most  appropriate  response  to  each  of  the  following 
statements. 

15.1  The  selection  official  has  sufficient  software  technical  expertise 
to  select  a  qualified  contractor. 

15.2  The  software-related  contractual  requirements  baseline  is 
established  prior  to  release  of  the  solicitation  package. 

1 5.3  The  solicitation  package  includes  the  contractual  software 
requirements  and  proposal  evaluation  criteria. 

1 5.4  Technical  reviewers  use  proposal  evaluation  criteria  during 
solicitation  activities. 

1 5.5  Software  risks  are  independently  evaluated  as  part  of  the 
solicitation  and  are  communicated  to  the  solicitation  official. 
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Strongly  Disagree 


Appendix  -  Survey  Questionnaire 


Software-Related  Contractual  Requirements  Development 

and  Management 

16.  The  following  items  address  whether  your  project  establishes  a 
common  and  unambiguous  definition  of  software-related 
contractual  requirements  that  is  understood  by  the  acquisition 
project  team,  end  user,  and  the  contractor  team. 

Indicate  the  most  appropriate  response  to  each  of  the  following 
statements. 

1 6.1  Software-related  contractual  requirements  are  developed, 
managed,  and  maintained  using  a  structured  process. 

1 6.2  End  users  and  other  affected  groups  have  input  to  the 
software-related  contractual  requirements  over  the  life  of  the 
acquisition. 

1 6.3  A  member  of  the  acquisition  project  staff  or  a  novice  engineer 
could  identify  and  verify  the  source  of  software-related 
contractual  requirements. 

1 6.4  In  the  case  of  new  and/or  changing  program  requirements, 
acquisition  project  staff  know  when  and  how  to  make  changes 
to  contractual  requirements,  including  acceptance  criteria. 

1 6.5  A  formal  control  board  is  in  place  to  authorize  changes  to 
requirements. 


0 

o 

<D 

z 

O) 

9? 

D) 

< 

CD 

CO 

(/) 

a 

O 

1 

D) 

a 

o 

0) 

CD 

kw* 

CD 

k— 

D) 

CO 

D) 

C 

o 

c 

.4—* 

"c 

O) 

C/) 

•4-I 

o 

CO 

< 

b 

(S) 

Q 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 
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Contract  Performance  Management 

17.  The  following  items  address  whether  your  project  team  ensures  that 
the  software  activities  under  contract  are  being  performed  in 
accordance  with  contractual  requirements. 

Indicate  the  most  appropriate  response  to  each  of  the  following 
statements. 

1 7.1  The  project  team  has  sufficient  insight  into  the  contractor’s 
software  engineering  effort  to  ensure  the  effort  is  managed  and 
controlled  and  complies  with  contract  requirements. 

1 7.2  The  acquisition  project  team  and  contractor  team  maintains 
ongoing  communication  and  both  parties  agree  to 
commitments. 

17.3  Your  project  team  identifies,  documents,  and  tracks  software 
risks  associated  with  the  contractor’s  efforts,  independent  of  the 
contractor’s  risk  management  process. 

1 7.4  The  quality  of  the  contractor  team’s  process,  performance, 
products,  and  services  are  appraised  throughout  the  contract’s 
period  of  performance  to  identify  risks  and  take  action  to 
mitigate  those  risks  as  early  as  possible. 
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Transition  to  Support 

0 

< 

18.  The  following  items  address  whether  projects  provide  for  the 
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transition  of  the  software  products  being  acquired  to  the  eventual 
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Indicate  the  most  appropriate  response  to  each  of  the  following 
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1 8.1  The  acquisition  project  team  ensures  that  the  software  support 

□ 

□ 

□ 

□ 

□ 

organization  has  the  capacity  and  capability  to  provide  the 
required  support  upon  assumption  of  responsibility  for  the 
support  of  the  software  products. 

18.2  The  acquisition  project  team  ensures  there  is  no  loss  in 

□ 

□ 

□ 

□ 

□ 

continuity  of  support  to  the  software  products  during  transition 
from  the  development  contractor  to  the  software  support 
organization. 

1 8.3  Configuration  management  of  the  software  products  is 

□ 

□ 

□ 

□ 

□ 

maintained  throughout  the  transition. 

1 8.4  The  strategy  for  transition  into  maintenance  is  documented. 

□ 

□ 

□ 

□ 

communicated,  and  agreed  to  by  all  parties  early  in  the 
acquisition. 

Transition  to  Operations 

19.  The  following  items  address  whether  acquisition  projects  prepare 
for  the  transition  of  the  software  products  being  acquired  to  the  end 
user. 

Indicate  the  most  appropriate  response  to  each  of  the  following 
statements. 


1 9.1  The  acquisition  project  team  ensures  that  the  end  user  has  the 
training,  experience,  and  resources  to  accept  the  software 
products  into  operational  use. 

1 9.2  The  acquisition  project  team  plans  for  sufficient  contractor 
support  during  end-user  acceptance  testing. 

1 9.3  The  strategy  for  transition  into  operations  is  documented, 
communicated,  and  agreed  to  by  all  parties  in  the  acquisition. 

19.4  The  software  support  organization  participates  in  all  project  and 
technical  reviews. 
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Appendix  -  Survey  Questionnaire 


System  Developer’s  Processes 

20.  The  following  items  address  your  perceptions  of  the  impact  of  the 
contractor’s  work  processes  on  the  success  of  your  software¬ 
intensive  system  acquisitions.  Please  rate  the  effectiveness  of  your 
contractor’s  abilities  in  the  following  areas. 

20.1  Project  planning 

20.2  Project  monitoring  and  control 

20.3  Requirements  development 

20.4  Requirements  management 

20.5  Measurement  analysis  and  reporting 

20.6  Software  architecture  development  and  assessment 

20.7  Technical  solution 

20.8  Product  integration 

20.9  Configuration  Management 

20.10  Risk  management 

20.11  Verification 

20.12  Validation 

20.1 3  Supplier  or  subcontract  management 

20.14  Integrated  teaming 

20.1 5  Defined  processes  that  support  product  development  and 
stable  operations 

20.16  Shares  all  relevant  information  that  you  feel  you  should  know  to 
manage  the  acquisition  effectively 

20.1 7  Causal  analysis  and  resolution  for  defect  prevention 
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Appendix  -  Survey  Questionnaire 


Impact  of  External  Factors  on  Acquisition 

21 .  The  following  items  address  the  impact  of  external  factors  on  the 
successful  outcome  of  an  acquisition  program. 

Indicate  the  most  appropriate  response  to  each  of  the  following 
statements. 


21 .1  Mandates  from  congress  inhibit  our  program  from  meeting  its 
goals. 

21 .2  Reallocation  of  program  funding  is  a  significant  source  of 
frustration  in  acquisition  programs. 

21 .3  Critical  personnel  are  lost  due  to  military  rotations  or  inability  to 
compete  with  industry  salaries. 

21 .4  Acquisition  reform  has  negatively  impacted  our  ability  to  meet 
our  objectives. 

21 .5  Expressions  of  user  requirements  throughout  the  acquisition 
process  causes  disruption  in  the  development  process. 

21 .6  Lack  of  test  bed  assets  to  stress  test  system  under  realistic 
operational  conditions  is  a  major  problem. 


Strongly  Agree 

Agree 

Disagree 

Strongly  Disagree 

Don’t  know  or  N/A 

□ 

□ 

□ 

□ 

□ 

□ 
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□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Where  are  the  major  problems  and  risks? 

22.  Indicate  your  perception  of  the  extent  to  which  each  of  the  following 
areas  (22.1-22.4)  contribute  to  major  problems  and  risks  in  the 
successful  outcome  of  acquisition  programs. 

When  responding,  please  consider  your  cumulative  experience  in 
acquisition  over  the  last  1 0  years. 

22.1  Acquisition  program  policies  and  processes 

22.2  The  contracting  process  between  acquirer  and  developers 

22.3  The  contractor’s  (or  suppliers)  development  process 

22.4  Factors  outside  the  control  of  the  acquirers  and  developers 
(e.g.,  congressional  mandates;  priorities  set  by  engagements 
of  our  armed  forces,  etc.) 
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23.  Overall,  what  are  the  one  or  two  most  difficult  problems  that  your  project  has  faced  in  conducting 
successful  acquisition  of  software-intensive  systems.  (Please  describe) 


24.  If  you  could  change  one  or  two  things  about  how  software-intensive  systems  are  acquired  in  the 
Army,  what  would  they  be?  (Please  describe) 
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